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This toolkit is designed to help school- and district-based practitioners engage in a continuous 
improvement effort. It provides an overview of continuous improvement and focuses on the Plan-Do- 
Study-Act cycle. It includes tools and resources that practitioners can use to implement a continuous 
improvement effort in their own schools, districts, or agencies. 
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INTRODUCTION TO THE TOOLKIT 


Continuous improvement is based on the principles that making sustainable change takes time and involves 
collective effort; is context specific; and requires constant adaptation, data collection, and learning (Bessant & 
Caffyn, 1997; Bhuiyan & Baghel, 2005; Langley, Moen, Nolan, Nolan, & Norman, 2009). Continuous improvement 
engages key players in a system to focus on a specific problem of practice and, through a series of iterative cycles, 
identify and test change practices (new approaches, tools, or strategies used to address the problem), make pre- 
dictions, collect data about the change practices, and study the potential influence of those change practices on 
outcomes of interest (figure 1; Bryk, Gomez, Grunow, & LeMahieu, 2015). Through these cycles the members of 
the system build their capacity to test proposed change practices; refine those change practices based on evi- 
dence; and increase the scale, scope, and spread of a change practice over time (Langley et al., 2009). 


Continuous improvement is premised on three core principles about how sustainable change is achieved: 
e Change takes time and involves collective effort (Bryk et al., 2015; Katz, Earl, & Jaafar, 2009). 


e Change is context-specific and therefore requires constant adaptation, data collection, and learning (Bryk 
et al., 2015). 


e Focusing on a series of small changes, combined with ongoing evidence collection and review, can lead to 
large-scale change (Derrick-Mills, Sandstrom, Pettijohn, Fyffe, & Koulish, 2014; Hawley, 2006; Park, Hironaka, 
Carver, & Nordstrum, 2013; Snow, Dismuke, Zenkert, & Loffer, 2017). 


Figure 1. The continuous improvement effort 
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The model for improvement consists of three main questions: 


What problem are we What change might How will we know that a change 
trying to solve? we introduce and why? is actually an improvement? 


Plan-Do-Study-Act (PDSA) Cycle 


1. PLAN 


Select a change practice to test. 
Select measures and develop a data 
collection plan. 


2.DO 


Implement the change practice. 
Collect data to inform improvement. 


4. ACT 


Based on data study, make improvements 
to the change practice, take steps to scale the 
change practice, and/or choose to try another 
change practice. 


3. STUDY 


Collectively examine data to inform 
improvement. 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 


Source: Authors’ creation. 


This toolkit is designed to orient school principals, district staff, teachers, and other practitioners to the principles 
and practices of continuous improvement. It draws on the Model for Improvement (Langley et al., 2009) and 
employs Plan-Do-Study-Act (PDSA) cycles to implement and test change practices. The overarching goals of the 
toolkit are to: 


e Build the capacity of education practitioners to use continuous improvement as a systematic approach for 
identifying areas for improvement, implementing intended improvements, collecting data related to imple- 
mentation, studying those data, and using the evidence in decisionmaking. 


e Provide accessible, practitioner-friendly guidance and tools to help education practitioners apply continuous 
improvement in their schools or school districts. 


It is not designed as a facilitator’s guide but is meant to be available in its entirety to everyone participating in the 
improvement effort. 


Intended users 


This toolkit is intended for a team of education practitioners (an Improvement Team) to use throughout a con- 
tinuous improvement effort. Additional users might include state-level education agency staff who are leading 
a continuous improvement effort with a group of schools or districts or in their own organization. The Improve- 
ment Team may include as few as three practitioners or as many as a dozen. If the team is much larger, engaging 
in productive conversations can become difficult. But there is no hard and fast rule about the right number of 
participants, and users can adapt the practices in the toolkit to meet the needs of different group sizes and con- 
texts. Other key participants may include individuals who have knowledge of the school’s or district’s data and 
familiarity with how to display and use those data, as well as individuals with content knowledge who can provide 
expertise on the topic of interest. However, a content expert is not necessary for the team to begin its work. 
Rather, the appropriate expertise may be accessed once the team decides what to focus on. Part | provides more 
detail about the roles and responsibilities of Improvement Team members. 


Why continuous improvement? 


The continuous improvement processes described in this toolkit can help education practitioners agree on spe- 
cific challenges they face, identify change practices that can address those challenges, implement those change 
practices, study their implementation and outcomes, and decide whether the change practices are worthwhile 
and should be implemented or scaled up in their specific contexts. Most of the continuous improvement methods 
in use today stem from work by W. Edward Deming and others in industry in the mid-20th century (Deming, 
1993) and remain prevalent in manufacturing and other industries to improve products, processes, and services 
(Bhuiyan & Baghel, 2005). Continuous improvement approaches have also been widely used in healthcare for 
decades (Berwick, 1989; Kenney, 2008; Marjoua & Bozic, 2012). More recently, education and human service 
areas such as child welfare, education, early care and education, behavioral health, and public health have turned 
to continuous improvement approaches to improve outcomes for children, families, and adults (Derrick-Mills, 
2015; Fryer, Antony, & Douglas, 2007; National Child Welfare Resource Center for Organizational Improvement 
and Casey Family Programs, 2005; U.S. Department of Health and Human Services, 2011). In education, improve- 
ment science has been used, for example, to improve college credit accumulation in math and to reduce teacher 
turnover (Bryk et al., 2015). 


The value of continuous improvement—and the challenges 
and limitations 


While continuous improvement holds promise for supporting change in education, there is limited empirical evi- 
dence on the outcomes of continuous improvement in education. However, a few studies have attempted to 
examine the relationship between continuous improvement approaches, such as the PDSA cycle, and outcomes 
in medicine. One such study, a randomized controlled trial conducted in primary care facilities in the Netherlands, 
found that facilities that employed continuous improvement practices undertook and achieved more of their own 
improvement goals than did facilities that did not use the same systematic processes (Engels et al., 2006). This is 
a positive finding, but more rigorous research is needed that examines the efficacy of the approach in general and 
specifically in improving outcomes for schools and students. 


A review of 73 empirical studies in medicine found that the research on continuous improvement reveals more 
about the lack of fidelity to formal continuous improvement approaches than about the outcomes of continuous 
improvement efforts themselves (Taylor et al., 2013). Only 47 of the studies documented the continuous improve- 
ment process in enough detail to warrant analysis, and only 20 percent of those documented multiple cycles of 
iterative change. The review concluded that key principles of the PDSA cycles are rarely adhered to and that there 
is much room for improvement in applying and using that process. 


The review and other research suggest that across disciplines the success of continuous improvement efforts 
depends on several factors: organizational leadership that understands how to integrate continuous improve- 
ment across multiple levels (Robbins & Finley, 1995; Shortell, Bennett, & Byck, 1998), effective process manage- 
ment and staff engagement (Fryer et al., 2007), training and resources focused on disciplined approaches to 
implementation and learning by doing (Bryk et al., 2015), and increased commitment and capacity to regularly 
collect and use data (Derrick-Mills et al., 2014; Langley et al., 2009). 


Overview of the toolkit 


Part | of this toolkit provides guidance for the Continuous Improvement (Cl) leader and steps to take before 
embarking on a continuous improvement effort. Part Il provides content (readings, tools and templates, and 
videos) for a series of meetings that guide an Improvement Team through the process of identifying a common 
problem, generating a series of evidence-based change practices to test and study, testing those change practic- 
es, collecting and analyzing data, and reflecting on and using evidence to identify next steps. 


How the toolkit should be used 


Rather than reading the entire toolkit before the process begins, the Improvement Team should progress through 
the toolkit during implementation of a continuous improvement effort. It is useful to have a leader to guide the 
group and ensure that the appropriate materials and content are available for each meeting. The leader can 
be a principal, an administrator, a curriculum coordinator, a school counselor, a teacher, or any another team 
member. This person is referred to as the “Cl leader,” but tasks designated to the Cl leader can be completed by 
another member or members of the team. It is also possible for multiple people to share the Cl leader role. The Cl 
leader—along with a core group of participants, which may include a school leader, a teacher leader, and possibly 
one or two additional stakeholders—should read and engage in Part | and develop a familiarity with the process 
described in Part II before bringing together the team. The content of Part |, which consists of five steps described 
below, is designed to ensure that the key stakeholders are ready to engage in the continuous improvement effort, 
that all relevant stakeholders are engaged, and that barriers to success are addressed before embarking on the 


effort. The activities are designed for the core group, including the Cl leader and select other participants, after it 
has considered the school’s, district’s, or organization’s readiness to follow through on a continuous improvement 
effort. 


Part Il is designed so that Improvement Team members read the appropriate information as they proceed through 
the continuous improvement effort together, meeting by meeting. The meetings guide the team through the con- 
tinuous improvement effort and a series of PDSA cycles. Each meeting is presented in a similar way: a list of what 
team members need to do to prepare for the meeting, a list of materials needed for the meeting, a pre-meeting 
reading, a proposed meeting agenda, details of the meeting activities, and suggested next steps. Most meetings 
should last approximately 90 minutes; two meetings should last approximately 120 minutes. 


The toolkit provides a step-by-step process for an Improvement Team to follow over the course of the school 
year, beginning in the summer and continuing until the end of the school year. However, when the process begins 
is flexible, as is how long it lasts. The toolkit offers some guidance about the calendar and the number of meet- 
ings, but these are recommendations, not requirements. In general, the team should use the toolkit as is appro- 
priate to its needs. 


Grove Elementary School example case 


The fictional Grove Elementary School is used as an example case throughout the toolkit to illustrate key points. 
It provides the context for completed examples of several activities, such as the fishbone diagram template and 
the PDSA planning template. The case is described in the pre-meeting reading for Meeting 1 and Meeting 2 and 
through video and activities. Meeting 1 also presents other scenarios that demonstrate how continuous improve- 
ment might be relevant in different contexts and with different problems. 


Videos 


There are five videos that accompany the toolkit. Links to these videos are provided in the toolkit. Each video is 
three to four minutes long and is designed to be used during the meetings to prompt the group during an activity 
or to initiate a discussion on a specific topic. The videos integrate the Grove Elementary School example case 
referenced above. 


Additional tools 


The toolkit includes 10 templates and flow charts to help practitioners engage in continuous improvement. These 
are a template for a continuous improvement calendar; an infographic of the continuous improvement effort; a 
fishbone diagram template and completed example; a driver diagram template and completed example; a PDSA 
planning tool template and completed example; an Excel-based run chart template to support the presentation 
of data; an adapt, adopt, or abandon flow chart to guide decisions about next steps in implementing change prac- 
tices; a template for outlining a change practice; a template for communicating about a change practice and a 
completed example; and a template for implementation planning. These materials are provided as screenshots in 
the toolkit and are available for download at https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4591. 


PART |. PLANNING A CONTINUOUS 
IMPROVEMENT EFFORT 


Before an Improvement Team comes together to embark on a contin- Materials needed 

uous improvement effort, it is important for the Cl leader, along with a Handout 1—Checklist for assessing 

few other core participants, to preview the content of the toolkit and to readiness for a continuous improve- 

have a general understanding of the elements of continuous improve- ment effort (p. |-5) 

ment and Plan-Do-Study-Act (PDSA) cycles. This section outlines five Handout 2—Preliminary data 

steps for the Cl leader and the few additional core participants to take inventory worksheet (p. I-8) 

before the first meeting: Handout 3—Checklist for recruit- 
ing Improvement Team members 

e Step 1: Assess the school’s or district’s readiness to engage in a contin- (p. I-11) 

uous improvement effort. Handout 4—Improvement Team 

member roles and responsibilities 

e Step 2: Determine the overall focus of the continuous improvement (p. 1-13) 

effort. Handout 5—Continuous improve- 

ment calendar (p. I-16 and 

e Step 3: Recruit Improvement Team members. https://ies.ed.gov/ncee/edlabs/ 
regions/northeast/pdf/handout5__ 

e Step 4: Identify Improvement Team member roles and responsibilities. Cl-calendar.pdf) 
Handout 6—The continuous 

e Step 5: Plan the Improvement Team calendar. improvement cycle (p. I-17 and 
https://ies.ed.gov/ncee/edlabs/ 

Steps 1-3 may happen more or less concurrently. They are described regions/northeast/pdf/handout6_ 


here as discrete steps, but they inform one another. For example, while Cl-cycle.pdf) 
assessing the readiness of the school or district is necessary before 
moving forward, the Cl leader will have likely identified a focus while assessing readiness. The main idea is to 
engage in the preparations described here before the first meeting. 


Step 1. Assess the school’s or district’s readiness to engage in 
a continuous improvement effort 


To build the foundation for a continuous improvement effort, it is important to take stock of school or district 
resources and of educators’ readiness to make meaningful improvement toward achieving the goal and identify 
where capacity needs to be built. This step should be conducted by the Cl leader but should involve other stake- 
holders, such as relevant school and teacher leaders. Readiness should be assessed using the following seven 
questions, which are also described in Handout 1—Checklist for assessing readiness for a continuous improvement 


effort (p. |-5). 


A ere roa agreement on the need to change practice in the school or 
istrict: 


Broad agreement or critical mass on the need for change is necessary to create momentum for a continuous 
improvement effort. In one school a critical mass may mean that almost all the veteran teachers agree on a focus 
that provides impetus for the rest of the staff to get on board. In another school it may be just the principal, assis- 
tant principals, and instructional coaches who need to agree. The specific climate of the school will determine 
who needs to be on board for the continuous improvement effort to be successful. 


Are there enough ideas about how to respond to the need for change? 


Agreeing that change is needed is important but not sufficient to ensure readiness. It is also important for partic- 
ipants to have ideas about how to respond to the need for change. Have school leaders and teachers discussed 
the issue or problem previously, and do they have ideas about how to address the issue, based on research or 
knowledge of successful efforts elsewhere? Have the right participants engaged in those discussions? While the 
Improvement Team will identify practices or processes to implement as part of its formal process, it is useful to 
take some time during the planning phase to ensure that there are relevant ideas, practices, or processes to draw 
on in developing the practices the Improvement Team will implement as part of the continuous improvement 
effort. 


Does the school or district have a culture of collaboration, learning, and 
change? 


Ask these questions about the school’s or school district’s culture: 


e How does the work get done (or not get done) in the school or district? Some schools and districts are very 
top-down in management style, while others use teacher teams and teacher leaders to engage in distributed 
leadership. Structures that enable staff to work together and foster a culture of collaboration are important for 
a successful continuous improvement effort. 


e What are the prevailing attitudes toward change? Attitudes about change can range from fatigue and apathy, 
resulting from constant change and efforts that do not deliver results, to empowerment and interest in pur- 
suing new ideas. For a continuous improvement effort to succeed, participants should be open to examining a 
problem from different angles and be willing to change practices to address the problem. 


While a school or district culture that already supports collaborative work can contribute to the potential for 
success of a continuous improvement effort, engaging in continuous improvement can also help build the kind of 
collaborative culture described here. 


Does the leadership support the continuous improvement effort? 


The types of leadership support that are needed vary depending on the situation. If the Cl leader is a teacher, 
assessing leadership support likely requires talking to a principal or administrator. If the Cl leader is a principal, 
leadership support from the district office may be necessary. When assessing leadership support, consider not 
only general encouragement to engage in the continuous improvement effort but also more concrete support. 
Are relevant leaders prepared to provide the resources necessary for the effort to be successful? For example, 
will they not only encourage the work but also provide the staff time necessary to implement the effort? See 
below for more on the types of resources needed. 


Are there adequate resources to support the continuous improvement effort? 


The Improvement Team will need resources to engage in the continuous improvement effort. Key resources 
are time to fully engage in the process and money or other resources related to supplies and staffing. This may 
include time on the professional development calendar; space for the Improvement Team to work; and schedule 
flexibility to allow team members to visit one another’s classrooms, secure substitutes, or engage in other work 
beyond the traditional scope of their day-to-day responsibilities. In the absence of adequate resources, school 
or district practitioners who are interested in implementing a continuous improvement effort, particularly those 
with authority to reallocate resources, should work toward creating the structures or systems needed to engage 
in continuous improvement. 


Do potential Improvement Team members have basic data literacy skills, 
and are appropriate data available or accessible? 


Collecting, reviewing, interpreting, and acting on data are integral parts of continuous improvement. Assess 
potential Improvement Team members’ skills, experience, and confidence in collecting, examining, and using 
data, then determine whether training or practice opportunities are needed to build their data literacy skills. 
Sources of data and uses of data are covered in detail later in this toolkit. Focus on the availability of data that the 
team will collect later rather than on the data themselves. Step 2 provides more information on conducting an 
inventory of available data. 


What are potential barriers to successful implementation of the continuous 


improvement effort? 


Take stock of the anticipated or possible barriers to change that 
exist in the school or district. It is important to anticipate the orga- 
nizational, social, and resource obstacles that the Improvement 
Team may encounter throughout the process so that it can take 
these factors into account as it moves forward with the continu- 
ous improvement effort. 


Use Handout 1—Checklist for assessing readiness for a continuous 
improvement effort (see next page) to help determine whether the 
school or district is prepared to engage in a continuous improve- 
ment effort and to identify where capacity needs to be built to 
fully benefit from the work. 


How ready is the Improvement Team 
to engage with data? 


Data literacy is a collection of skills and 
knowledge for working with data to inform 
decisionmaking. Data literacy includes effec- 
tively collecting, organizing, presenting, inter- 
preting, summarizing, and critiquing data in 
ways that result in actionable understanding 
(Gummer & Mandinach, 2015; Mandinach & 
Gummer, 2013). 

Consider what skills the team has in data 
terminology, data context (that is, who, when, 
and how data are collected), and critical data 
review. If they need to develop additional 
skills, consider including selections from the 
following resources as homework before or 
between meetings: 

Bocala, C., Henry, S. F., Mundry, S., & 
Morgan, C. (2014). Practitioner data use in 
schools: Workshop toolkit (REL 2015-043). 
Washington, DC: U.S. Department of Educa- 
tion, Institute of Education Sciences, National 
Center for Education Evaluation and Regional 
Assistance, Regional Educational Laboratory 
Northeast & Islands. 

McEwan, E. K., & McEwan, P. J. (2003). 
Making sense of research: What’s good, 
what’s not, and how to tell the difference. 
Thousand Oaks, CA: Corwin Press. 

Popham, W. J. (2010). Everything school 
leaders need to know about assessment. 
Thousand Oaks, CA: Corwin Press. 


Handout 1. Checklist for assessing readiness for a continuous 
improvement effort 


Use this checklist to help determine whether your school or district is prepared to begin a continuous improve- 
ment effort and what you may need in order to get ready. Mark each checkbox where you and, if appropriate, 
other relevant stakeholders agree that your school or district meets the criterion. Consider convening a small 
group to discuss and assess readiness according to these seven criteria. If a few boxes remain unchecked, focus 
on building readiness in these areas either prior to convening the Improvement Team or as the team’s first tasks. 
If many boxes remain unchecked, work with relevant stakeholders to build readiness prior to beginning a contin- 
uous improvement effort. As you consider the readiness topics, keep these questions in mind and take notes as 
appropriate: On what evidence can we base these answers? What additional information do we need to collect? 
How might we adjust our resources, culture, timeline, approach, or activities to increase readiness? 


e Broad agreement on the need to change practice in your school or district 
There have been discussions in the school or district about the overall focus. 
A critical mass of people agree that change is necessary to address the overall focus. 


e Enough ideas about how to respond to the need for change 

Participants have ideas about how to respond to the issue or problem that has been identified. 

There are examples of practices and processes, from research or knowledge of successful efforts else- 
where, that may be applicable to address the issue or problem. 


e School or district culture of collaboration, learning, and change 

Structures are in place for staff to collaborate in the school, or a culture of collaboration exists. 
There is a culture of learning and openness to change. 

There is interest and willingness among participants to engage in continuous improvement. 


e Leadership support for the continuous improvement effort 

The principal or other school or district leaders are aware of the proposed continuous improvement 
effort. 

Leadership is willing to provide time and space for the Improvement Team to engage in the continuous 
improvement effort. 


e Adequate resources to support the continuous improvement effort 

Resources have been identified (such as release time, meeting space, or personnel). 

Improvement Team members’ schedules allow for meeting and working on the continuous improvement 
effort. 

Resources needed: 


e Basic data literacy skills of potential Improvement Team members and availability and accessibility of appropri- 
ate data 

Sufficient data on the overall focus can be collected. 

The Improvement Team is comfortable and experienced working with data. 


e Potential barriers to successful implementation of the continuous improvement effort 
Barriers that might hinder the continuous improvement process: 


Notes: 


Step 2. Determine the overall focus of the continuous 
improvement effort 


Before building an Improvement Team to engage in a continuous improvement effort, it is important to identify 
a general issue, or overall focus, for the work. Educators using this toolkit probably have particular interests or 
concerns that they believe are appropriate to address through a continuous improvement effort. Because engag- 
ing in continuous improvement processes can be time consuming, having an overall focus from the beginning will 
help the team move forward with the continuous improvement effort and get to implementing the change prac- 
tices more quickly. An example of an overall focus is math achievement or school climate. The overall focus helps 
direct and define the continuous improvement effort but is not too specific or narrow a problem and related 
aim, both of which the team will define in early meetings. The overall focus also helps determine the appropriate 
people to include on the team, such as a district math coach if the overall focus will be math achievement. 


Two elements are important in determining the overall focus: examining relevant data and soliciting input from 
multiple stakeholders. 


Examining relevant data 


A Cl leader and core team considering implementing a continuous improvement effort likely already have an 
idea for an overall focus, which has led them to consider continuous improvement. For example, the Cl leader 
and core team may derive the initial focus based on the school or district strategic plan, goals, or prior issues. 
They may have already looked at data that have informed the determination of an overall focus. However, the 
continuous improvement effort should be guided by data, so the Cl leader and core team should take time to 
identify and examine relevant data that will help define the overall focus of the continuous improvement effort. 
It may be useful to conduct a preliminary data inventory (see Handout 2—Preliminary data inventory worksheet 
on p. I-8) to identify existing data related to the overall focus of the continuous improvement effort. Having a 
preliminary understanding of existing data as well as of the baseline from which the Improvement Team will be 
working, such as the percentage of students who score proficient on a state assessment or the average number 
of student absences each month, will help in defining and communicating the overall focus. 


There will be more opportunities to collect and examine data after the first meeting, but at this point in the 
process the goal is for the Cl leader to be able to prepare a brief statement (a few sentences) about the overall 
focus of the continuous improvement effort to share with potential members of the Improvement Team as part 
of the recruitment process. Once the team begins to meet, the issue to address will become more specific, as 
the team discusses the overall focus and reviews relevant data. Below is an example of a brief statement related 
to the Grove Elementary School example case. The overall focus does not need to reference specific data, but 
the statement should be grounded in data; in other words the Cl leader should be able to back up the statement 
based on the initial review of the data. These data will be important as the team crafts the problem and sets 
the aim. In addition, the data will be important in determining whether the changes the team implements yield 
improvements related to the problem or issue they have identified. However at this stage, identifying elementary 
math achievement as the issue to address is enough for recruiting team members. 


OVERALL FOCUS FOR GROVE ELEMENTARY SCHOOL: Student achievement in elementary math is an 
area of concern, based on recent state achievement results. Our continuous improvement effort will 
focus on this issue. 


Soliciting input from multiple stakeholders 


The Cl leader should also engage in conversations with key stakeholders who represent multiple perspectives. 
These conversations will help the Cl leader discern readiness to engage in continuous improvement and ensure 
that enthusiasm and potential for change exist for the overall focus. For example, when speaking to teachers 
about elementary math achievement, the Cl leader may discover that teachers believe the issue is related to the 
implementation of a new math curriculum and the teachers’ and students’ capacity to engage with it. This discov- 
ery may lead the Improvement Team to define the overall focus more specifically on the math curriculum while 
keeping the state assessment results an outcome of interest. 


Complete Handout 2—Preliminary data inventory worksheet (see next page) to identify existing data related to 
the overall focus of the continuous improvement effort. 


Handout 2. Preliminary data inventory worksheet 


Use the worksheet on the next page to list sources of data that you currently have access to or that you wish to 
have access to in order to identify the overall focus of the continuous improvement effort and to support further 
data review with the Improvement Team. Consider a variety of data sources, such as: 


Standardized assessments. For example, state or district assessments, Advanced Placement tests, college 
admissions tests, and standardized progress monitoring assessments. 


Other student achievement or progress measures. For example, formative assessments, end-of-unit tests, 
teacher-developed assessments, student work samples, screening and progress monitoring measures, and 
class reading lists. 


Nonacademic student data. For example, attendance data, discipline referrals, retention records, behavioral 
assessments, and student class schedules. 


Data on curricula and instruction. For example, classroom observations, lesson plans, and teacher journals or 
logs. 


Other data sources. For example, school climate surveys, data from professional development sessions, and 
interviews with or surveys of families, teachers, and administrators. 


Use this worksheet to identify relevant existing or needed data. For each source, consider: 


For whom do we have data? For example, for all grade levels, only grades 6-8, and so on. 


When and how often are the data collected? For example, literacy screening assessment scores for all students 
from September and then monthly only for students receiving literacy interventions. 


Where are the data located? For example, a student information system or a district database. 


How do we currently use these data? For example, scheduling students into classes or informing instruction. 


What relevant data do we already have or do we need? 


Yetxola]olucolaM (me)ann alo)antan'ulaarclam-larem ale) mo)anslem ae) |(cYouncle a 
Data source Where located? How do we currently use?) 


aie WViome) a al=Xave| 


LJ Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


L] Have 


Need 


Summarize: What relevant data do we already have or do we need? 


Step 3. Recruit Improvement Team members 


The recruitment stage enables the Cl leader to communicate to potential Improvement Team members the 
purpose of the work and to articulate expectations of participation. It is also an opportunity to collect impor- 
tant information from potential team members about their concerns, ideas, and interest to engage in the work. 
Handout 3—Checklist for recruiting Improvement Team members (see next page) outlines the steps to take in 
recruiting participants. 
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Handout 3. Checklist for recruiting Improvement Team 
members 


This checklist outlines the steps to take in recruiting participants for the Improvement Team. As a guideline, the 
team should be large enough that multiple perspectives are represented but small enough that group discussions 
are feasible and productive. Team members should include people who will actually test the change practices the 
team identifies. 


_____ Identify potential Improvement Team members. 
e Who cares about the overall focus? 
e Who is affected by the overall focus? 
e Who can positively affect the overall focus? 


e How can we encourage diverse perspectives on the overall focus (veteran and new teachers; general and 
special education teachers; administrators and teachers; parents, students, or community members)? 


e Who will attend and actively participate in the meetings? 

e Who will actively participate in collecting data and testing change practices? 

Prepare for conversations with potential Improvement Team members. 

e Fillin and print Handout 4—Improvement Team member roles and responsibilities (p. |-13). 


e Be prepared to briefly describe, in three to four sentences, the overall focus to be addressed, including 
sharing any related data. 


Have one-on-one or group conversations to recruit Improvement Team members. 

e Contact potential team members at least two weeks before the first meeting. 

e Explain the potential overall focus to be addressed. 

e Give potential team members an opportunity to react to and provide input on the overall focus area. 
e Share the goals of the continuous improvement effort. 


e Set expectations for participation (share Handout 4—Improvement Team member roles and responsibili- 
ties (p. I-13). 


e Identify and discuss any barriers to the person’s participation (scheduling, workload, interest, supervisor 
support) and how to overcome them. 


e Based on this discussion, if the person appears ready, willing, and interested, ask him or her to commit to 
attending the meetings and participating in the effort. 


Explain that specific roles for Improvement Team members will be designated during the first meeting. 


I-11 


Step 4. Identify Improvement Team member roles and 
responsibilities 


All Improvement Team members will participate in planning and testing potential change practices, and some will 
fill important roles during and between meetings. Team members should understand what they are committing 
to and what role they will play in the continuous improvement effort. Assigning specific roles for team members 
ensures that the work is distributed and that no one person is overly burdened. The work includes everything 
from scheduling and facilitating meetings to managing the data to securing a meeting space. When identifying 
and recruiting team members, think about the specific tasks and related skills that are needed and intentionally 
recruit individuals who have the capacity and interest to take on specific roles. These roles include: 


e Cl leader. The continuous improvement effort will be most successful when there is an individual or small team 
of people who champion the work and ensure that work continues in the face of any challenges. The leader or 
leaders will do the initial recruitment, data or information collection, and other work to kick off the effort and 
will communicate with school or district leaders throughout the process. The leader also has an important role 
in maintaining the morale of the group and periodically helping the group celebrate progress. 


e¢ Meeting facilitator. A single person can facilitate all the meetings, or the work can be distributed such that a 
different person has responsibility for each meeting or a pair of people have responsibility for the full year. 
Whatever the approach, it is valuable to designate a meeting facilitator for each meeting. 


e Data manager. It is valuable to have one person responsible for consistently organizing and presenting the 
data that the Improvement Team collects during each PDSA cycle. This toolkit provides some resources to help 
with that responsibility, but consider who has the capacity, and ideally some data savvy (including knowledge 
of spreadsheets), to take on this responsibility. 


e Recorder. A lot of important discussion and documentation of ideas will need to be captured in each meeting. 
Designate one person or rotate responsibility for taking notes and storing them in a shared location so that 
ideas and decisions are documented consistently. 


Handout 4—Improvement Team member roles and responsibilities (see next page) highlights key roles and their 
corresponding responsibilities. The document is designed to be distributed to Improvement Team members as 
part of the recruitment process and to be shared during the first meeting so that everyone is aware of team 
member roles and responsibilities. 
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Handout 4. Improvement Team member roles and 
responsibilities 


This handout provides descriptions of the key roles and responsibilities of all Improvement Team members. 
All Improvement Team members 


Improvement Team members commit to engage in the continuous improvement processes, including Plan-Do- 
Study-Act (PDSA) cycles. The four-step PDSA process includes the initial phase of defining the problem and iden- 
tifying and generating a change practice or set of change practices (in other words, the particular processes or 
tools to adapt, modify, or implement), as well as testing the change practices in the classroom or school, col- 
lecting data, and studying the data to inform further modifications. The team will have diverse membership to 
ensure that varied perspectives support the change effort. 


Responsibilities of Improvement Team members include: 


e Participating in meetings (see calendar for dates, times, and locations). Regular participation is required. The 
team will set its goal for participation together. 


e Reviewing meeting materials before meetings. 
e Participating in a defining the problem activity. 
e Participating in defining the change practices to be implemented. 


¢ Committing to piloting the change practices in their own classroom or context or to supporting use of the 
change practices among appropriate educators. 


e Collecting data related to implementation in advance of data discussions, as needed. 


e Participating in discussions about data and implementation that will lead to decisions on whether to adopt, 
adapt, or abandon change practices. 


Continuous Improvement leader (CI leader) 


The Cl leader champions the effort and takes overall and consistent responsibility for the continuous improve- 
ment effort. Other team members may lead some of the tasks and facilitate meetings. 


Responsibilities of the Cl leader (with support of others, as appropriate) include: 
e Reviewing the full content of the toolkit before commencing work with the Improvement Team. 
e Assessing the school’s or district’s readiness to engage in the continuous improvement effort. 


e Recruiting participants and communicating directly with all participants regarding the project, including check- 
ing in on their progress. 


e Managing the calendar for the project, coordinating meetings, and facilitating (or assigning facilitators for) 
each meeting. 


1-13 


e Collecting preliminary data on the potential problem that the Improvement Team will address. 
© Communicating with other school- or district-based staff about the project. 
e Identifying and working with the data manager. 


e Identifying and securing any training or capacity building related to implementing the change practices that 
the Improvement Team decides to test. This may require outside resources or support. 


e Periodically allocating time throughout the continuous improvement effort to acknowledge and validate the 
progress that the Improvement Team is making. 


Meeting facilitator 


The meeting facilitator takes responsibility for one or all meetings (to be determined by the Improvement Team), 
including preparations for the meeting and management of the actual meeting. 


Responsibilities of the meeting facilitator include: 
e Reviewing the agenda and content of the meeting before it occurs. 


e Preparing any materials or activities required for the meeting. 


Ensuring that Improvement Team members have all materials required for the meeting. 


Coordinating and facilitating the meeting. 
Data manager 


The data manager is responsible for managing the data collected by Improvement Team members and related to 
implementation. 


Responsibilities of the data manager include: 
e Supporting the Cl leader in managing data collection. 


e Providing expertise related to data collection and analyses and building the capacity of Improvement Team 
members in data collection, analysis, and use. 


e Leading data analysis, with support from others, and working with the Cl leader to summarize and present data 
to the Improvement Team. 


Recorder 


The recorder is responsible for taking meeting notes and storing them in a shared location so that discussions and 
ideas are documented. 


Responsibilities of the recorder: 
e Taking notes during meetings. 
e Saving the notes in a timely manner in a shared location that is accessible to all Improvement Team members. 


e Keeping track of change practices and changes implemented. 
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Step 5. Plan the Improvement Team calendar 


At the outset of the continuous improvement effort it is useful to plan, to the extent possible, when and how the 
Improvement Team will meet. If the team members begin the year with all the meetings on their calendars, it will 
be easier to ensure that the meetings take place and that the work between meetings is accomplished. Although 
this toolkit provides materials and agendas for 10 meetings, the exact number of meetings will depend on the 
number of PDSA cycles the team undertakes. It is better to plan for more meetings and then cancel ones that are 
not needed. Also consider whether structures are already in place, such as professional development or common 
planning time sessions, that could be used for these meetings. 


Before the first meeting the Cl leader should outline the details of when and how often the group will meet 
using Handout 5—Continuous improvement calendar (see next page or https://ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout5_Cl-calendar.pdf, which includes an editable PDF). The team will probably want to final- 
ize meeting dates during the first meeting, but it would be helpful to enter potential meeting times before the 
first meeting so that they can be shared with the team. At the end of the first meeting the Improvement Team will 
discuss next steps and the calendar of meetings. 


Handout 6—The continuous improvement cycle, which follows Handout 5 (and is available at https://ies.ed.gov/ 


ncee/edlabs/regions/northeast/pdf/handout6_Cl-cycle.pdf) can be used to communicate both the basics of con- 
tinuous improvement and the specifics related to each step of the Improvement Team’s work. 
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Handout 5. Continuous improvement calendar 


SREL 


NORTHEAST 
& ISLANDS 


Regional Educcionl Laboratory 
ALEDC 


@oyalaialecolecminnl)cey-it-leimercl(salerle 


Many schools begin continuous improvement cycles in the summer, 
but it is possible to begin at any time of the year. 


Prepare for the continuous 


improvement effort 


« Assess school or district 
readiness 


* Recruit potential Improvement 


Team members 


¢ Define a general issue or topic 


to address 
* Inventory available data 


SUMMER 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 


Convene the improvement 
team 


Meeting 1: Introduction to continuous 
improvement 


Meeting 2: Defining the problem 
and determining root causes using a 
fishbone diagram 


Meeting 3: Establishing an aim and 
developing a driver diagram 


Meeting 4: Preparing for 
Plan-Do-Study-Act cycles 
Meeting 1 date: 

Meeting 2 date: 

Meeting 3 date: 

Meeting 4 date: 


FALL 


Begin PDSA cycles 


Meetings 5-8:* Examining results and 
related data 


*Depending on the number of PDSA cycles, the 
number of meetings to reflect on the PDSA 
cycles (Meetings 5-8 in the toolkit) will vary. 

Meeting 5 date: 
Meeting 6 date: 
Meeting 7 date: 


WINTER 


Continue PDSA cycles 


Meetings 5-8:* Examining results and 
related data 


Meeting 9: Reflecting across multiple 
Plan-Do-Study-Act cycles 


Meeting 10: Preparing to implement a 
change practice more broadly 


Meeting 8 date: 
Meeting 9 date: 
Meeting 10 date: 


SPRING 
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Handout 6. The continuous improvement cycle 


:R 


L , : 
nommeast COntinuous improvement cycle 


& 


The model for improvement consists of three main questions: 


AYAVA a¥eLimedae)o) (lan ielacm nie What change might How will we know that a change 


trying to solve? we introduce and why? is actually an improvement? 


Plan-Do-Study-Act (PDSA) Cycle 


1. PLAN 


Select a change practice to test. 
Select measures and develop a data 
collection plan. 


2.DO 


Implement the change practice. 
Collect data to inform improvement. 


4. ACT 3. STUDY 
Based on data study, make improvements Collectively examine data to inform 
to the change practice, take steps to scale the ACT improvement. 


change practice, and/or choose to try another 
change practice. 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 


1-17 


Review of actions to take to prepare for a continuous 
improvement effort 


e Assess readiness. Use Handout 1—Checklist for assessing readiness for a continuous improvement effort 
(p. 1-5). 


e Take an inventory of available data. Use Handout 2—Preliminary data inventory worksheet (p. |-8). Then 
prepare a brief statement about the overall focus of the effort, as in the example provided in step 2. The state- 
ment should be no more than a few sentences and should capture the overall topic the Improvement Team will 
address. 


e Recruit the Improvement Team. Use Handout 3—Checklist for recruiting Improvement Team members (p. |-11), 
and use Handout 4—Improvement Team member roles and responsibilities (p. |-13) to discuss with potential 
team members. 


e Fill in Handout 5—Continuous improvement calendar (p. |-16 and https://ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout5_Cl-calendar.pdf) to the extent possible. 


e Send Improvement Team members a PDF of this toolkit and ask them to complete the pre-meeting reading, 
which provides background on continuous improvement, offers an overview of the PDSA cycle, and introduc- 
es the Grove Elementary School example case that appears throughout the toolkit. Also ask them to review 
Handout 6—The continuous improvement cycle (p. I-17 and https://ies.ed.gov/ncee/edlabs/regions/northeast/ 
pdf/handout6_Cl-cycle.pdf). 


1-18 


PART Il. IMPROVEMENT TEAM MEETINGS 


This part of the toolkit provides content (readings, tools and templates, and videos) for a series of meetings 
that guide the Improvement Team through the process of identifying a common problem, generating a series of 
evidence-based change practices to test and study, testing those change practices, collecting and analyzing data, 
and reflecting on and using evidence to identify next steps. The toolkit provides materials and agendas for 10 
meetings, but the exact number will depend on the number of Plan-Do-Study-Act cycles the team undertakes. 
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Meeting 1. Introduction to continuous improvement 


This is the first meeting of the Improvement Team. It will orient team 


members to continuous improvement and Plan-Do-Study-Act (PDSA) Materials needed 

cycles (which are described in the pre-meeting reading and explained in Pre-meeting reading: Background 

more detail in Meeting 4), discuss the focus for the work, discuss roles on continuous improvement 

and responsibilities, and begin to plan the calendar. The team will also (p. II-3) 

watch the first video, which briefly tells the story of a group of teach- Handout 2—Preliminary data 

ers and their principal who collaboratively engaged in a continuous inventory worksheet (p. |-8) 

improvement effort. Handout 4—Improvement Team 

member roles and responsibilities 

Preparations (p. 1-13) 

Handout 5—Continuous improve- 

Cl leader ment calendar (p. |-16 and 

https://ies.ed.gov/ncee/edlabs/ 

e Send a PDF of the toolkit to Improvement Team members. regions/northeast/pdf/handout5 __ 

Cl-calendar.pdf) 

e Ask Improvement Team members to complete the pre-meeting Handout 6—The continuous 
reading (see next page), which provides background on continuous improvement cycle (p. I-17 and 
improvement and an overview of the PDSA cycle and introduces the https://ies.ed.gov/ncee/edlabs/ 
Grove Elementary School example case that appears throughout the regions/northeast/pdf/handout6_ 
toolkit, and to review Handout 6—The continuous improvement cycle Cl-cycle.pdf) 


(p. 1-17). Handout 7—Continuous improve- 


ment scenarios and guiding 
e Prepare a brief statement on the focus of the continuous improve- questions (p. II-9) 
ment effort. The statement should be no more than a few sentences. 


e Prepare Handout 4—Improvement Team member roles and responsibilities (p. I-13) and Handout 5—Continuous 
improvement calendar (p. I-16 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout5_Cl-calendar. 
pdf) for your specific context. 


e Assign someone to take notes if a recorder has not already been designated. 

Improvement Team members 

¢ Complete the pre-meeting reading (see next page), which provides background on continuous improvement 
and an overview of the PDSA cycle and introduces the Grove Elementary School example case that appears 


throughout the toolkit, and review Handout 6—The continuous improvement cycle (p. I-17 and https://ies. 
ed.gov/ncee/edlabs/regions/northeast/pdf/handout6_Cl-cycle.pdf). 
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Pre-meeting reading: Background on continuous 
improvement 


Overview of continuous improvement 


What is continuous improvement and why is it relevant for schools and districts? 
Continuous improvement is premised on three core principles about how sustainable change is achieved: 
e Change takes time and involves collective effort (Bryk et al., 2015; Katz, Earl, & Jaafar, 2009). 


e Change is context-specific and therefore requires constant adaptation, data collection, and learning (Bryk 
et al., 2015). 


e Focusing on a series of small changes, combined with ongoing evidence collection and review, can lead to 
large-scale change (Derrick-Mills et al., 2014; Hawley, 2006; Park, Hironaka, Carver, & Nordstrum, 2013; Snow, 
Dismuke, Zenkert, & Loffer, 2017). 


Continuous improvement processes engage key players in a system to focus on a specific problem of practice 
and, through a series of iterative cycles, identify and test change practices, make predictions, collect data about 
the change practices, and study the potential influence of those change practices on outcomes of interest, such 
as student attendance or math performance (Bryk et al., 2015). Through these cycles the members of the system 
build their capacity to test proposed change practices; refine those change practices based on evidence; and 
increase the scale, scope, and spread of a change effort over time (Langley et al., 2009). Different terms such 
as “improvement science,” “quality improvement,” and “performance management” are used to represent con- 
tinuous improvement philosophies and practices. Continuous improvement models and approaches include the 
Model for Improvement (Langley et al., 2009), Six Sigma (Harry & Schroeder, 2005; Kwak & Anbari, 2006), and 
Plan-Do-Study-Act (PDSA) cycles (Bryk et al., 2015), among others. What these approaches have in common is that 
they offer a systematic process through which practitioners can engage in identifying needs, defining the problem, 
implementing new or modified practices, collecting data, and reflecting. In addition, continuous improvement 
approaches can help practitioners implement existing evidence-based strategies by examining the results of the 
strategies at regular, short-term intervals in their own context. The continuous improvement process provides a 
set of tools and routines for adapting and refining evidence-based strategies to address local needs. 


Most of the continuous improvement methods in use today stem from work by W. Edward Deming and others 
in industry in the mid-20th century (Deming, 1993) and remain prevalent in manufacturing and other industries 
today to improve products, processes, and services (Bhuiyan & Baghel, 2005). Continuous improvement approach- 
es have also been widely used in healthcare for decades (Berwick, 1989; Kenney, 2008; Marjoua & Bozic, 2012). 
More recently, education and human service areas such as child welfare, education, early care and education, 
behavioral health, and public health have turned to continuous improvement approaches to improve outcomes 
for children, families, and adults (Derrick-Mills, 2015; Fryer et al., 2007; National Child Welfare Resource Center 
for Organizational Improvement and Casey Family Programs, 2005; U.S. Department of Health and Human Ser- 
vices, 2011). Continuous improvement approaches have been used in healthcare to improve customer service to 
patients, reduce medication errors, reduce the number of ventilator days for intensive care patients, and reduce 
patient waiting time (Langley et al., 2009). In education, improvement science has been used, for example, to 
improve college credit accumulation in math and to reduce teacher turnover (Bryk et al., 2015). 


While the research on the effectiveness of continuous improvement in education is limited, continuous improve- 
ment holds promise as a way to support schools and districts in engaging in systematic processes to support 
change practices that have the potential to improve teaching and learning. 
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What is the Model for Improvement? 


The Model for Improvement refers to the three essential ques- 
tions that guide the continuous improvement process:" 


e What problem are we trying to solve? For a school, district, or 
organization to improve, key participants must set clear and 
specific goals derived from a clear articulation of a problem 
or issue that requires attention. In defining the problem, par- 
ticipants identify an objective (the aim statement) that they 
intend to accomplish through the continuous improvement 
effort. The aim statement should target a specific population, 
be time specific, and be measurable. The aim statement goes 
beyond the overall focus that the Cl leader has identified as part 
of pre-planning and recruitment. An example is provided in the 
box to the right to illustrate the difference between the overall 
focus and problem statement. 


e What change might we introduce and why? Continuous improve- 
ment requires key participants to develop, test, and implement 
change practices. Selecting, testing, and implementing these 
change practices—for example, by trying out new protocols or 
processes—are at the core of continuous improvement. 


¢ How will we know that a change is an improvement? An essen- 
tial part of continuous improvement is examining whether the 
change practice has in fact addressed the problem and made 
some meaningful improvement. Clear and specific measures to 
capture both the processes and the outcomes are therefore at 
the heart of continuous improvement. As quality improvement 
experts often say, “Some is not a number, and soon is not a 
time” (Berwick, Calkins, McCannon, & Hackbarth, 2006). Thus, 
the Improvement Team sets measurable targets and then col- 
lects and analyzes data to determine whether progress is being 
made toward those targets. 


What is the right context for implementing continuous improvement? 


Key terms 


Overall focus. A brief statement of the broad 
issue the Improvement Team will address. It 
could be as general as a focus on elementary 
math achievement or more specific, based on 
a review of the data. Example: Student achieve- 
ment in elementary math is an area of concern, 
based on recent state achievement results. 


Problem statement. A brief statement of 
the specific and measurable problem that the 
Improvement Team will address. Example: 
Students in elementary grades are below pro- 
ficient in math, and low-income students are 
disproportionately represented among those 
students. 


Aim statement. The specific goal, devel- 
oped in response to the problem statement, 
that guides the improvement effort. It should 
describe what the Improvement Team wants 
to achieve and should be specific and mea- 
surable. Example: The percentage of students 
performing at proficient or above will increase 
by 10 percent, and the gap between low- 
income students and their peers will decrease. 


Change practices. The interventions or spe- 
cific practices selected to address the problem 
and achieve the aim. These interventions or 
practices will be tested during the continuous 
improvement effort. Example: Teachers use 
sentence starters to prompt math discourse. 


Continuous improvement can be used with a team of practitioners who have a common challenge, which can be 
quite broad, such as below grade-level math performance among all or a subgroup of students, or more specific, 
such as low student engagement in literature discussions. Employing the strategies that are part of continuous 
improvement can help the Improvement Team, which may include teachers, curriculum coordinators, content 
coaches, principals, and others, further refine the problem and generate ideas to address it. In a school or school 
district, educators face many problems for which continuous improvement may be appropriate. 


1. These three questions were adapted from the “Model for Improvement,” which was developed by Associates in Process Improve- 
ment and adapted by the Institute for Healthcare Improvement (Institute for Healthcare Improvement, 2015). 
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Plan-Do-Study-Act cycles 


While the questions in the Model for Improvement guide the overall continuous improvement effort, PDSA cycles 
provide the structure to support the developing, testing, and studying of change practices. The PDSA cycle is 
a four-step process that is useful in guiding continuous improvement to test a change practice in a real-world 
setting. Once the Improvement Team has defined the problem and set an aim, it can use the cycle to guide rapid 
learning through four steps that are repeated as part of an ongoing cycle of improvement: 


e Plan. After the Improvement Team has set an aim and generated possible change practices to test, the plan 
step provides the opportunity to further define a specific change practice to test and establish more specific 
targets or objectives. Such objectives may include formulating a theory about why the specific change practice 
selected might make a difference or hypothesizing about what outcomes are expected. During this step the 
team will also define the metrics for assessing implementation and outcomes of the specific change practice. 


e Do. This step involves implementing the plan and collecting data. During this step a specific change practice is 
tested, and data are collected. Some initial data analyses may also take place during this step. 


e Study. In this step participants use a guided protocol to examine the data and consider the extent to which the 
specific targets or objectives for the previous step were met. During this step the Improvement Team com- 
pares the data with the predictions or hypotheses put forward in the Plan step to see whether there are signs 
of progress toward the aim or problems or areas worthy of further consideration or modifications. 


e Act. This last step integrates all the learning that was generated throughout the process, which can be used to 
adjust the tools or processes being tested or to further the objectives or targets; to formulate new theories or 
predictions; and to make changes to the overarching aim. In this step participants may be ready to implement 
the change practice more broadly, or they may require another test in a different context or with other vari- 
ables. Then the cycle begins again. 


Often, multiple PDSA cycles are necessary to determine whether a change practice is resulting in improvement. 
Each cycle builds on the previous cycle as evidence builds about the implementation and outcomes of the change 
practices. 


Grove Elementary School example case: Part | 


The Grove Elementary School example case below will be used throughout the toolkit to illustrate key ideas and 
processes. In Meeting 1 you will hear from some practitioners in a video about their experience. The example case 
is based on the work of those practitioners, but some changes have been made to the story to illustrate key ideas 
presented throughout the toolkit. Consider the following questions as you read this case: What is the problem that 
Grove Elementary School faces? How do members of the school community know this is a problem? What could 
they do to address the problem? 


Grove Elementary School is in the second year of implementing a new curriculum and instructional strategies that 
are aligned with the Common Core Standards for Mathematical Practice, which define a range of skills that math 
educators should seek to develop in their students. The school recently received its state assessment data, and 
administrators noticed that math scores were lower than in prior years, particularly among low-income students. 
The principal has also observed that some teachers lack sufficient content or pedagogical knowledge to provide 
students with instruction that matches the Common Core Standards; she has observed this specifically in engag- 
ing students in math discourse. 
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The principal met with a group of teachers to discuss the lower math scores and the challenges associated with 
implementing the new curriculum and standards. The group discussed the standards for math practice and things 
that were and were not working in the classroom. The teachers agreed that what was working well is that stu- 
dents were displaying quantitative reasoning and could use a variety of materials to practice their skills. However, 
teachers were concerned that many of their students could neither construct viable arguments nor analyze sit- 
uations and communicate them to others. To better understand teachers’ concerns, the principal decided to 
observe some teachers during math lessons. 


The principal selected teachers at different grade levels to observe and used a simple protocol to track teacher 
questions and student responses. She noted that some students seemed to be less engaged during the lessons. 
The principal met again with the teachers and shared the results of the initial observations, focusing on what the 
teachers were doing and on the nature and frequency of student responses. During this meeting, after reviewing 
the results of the principal’s data collection, the Improvement Team more clearly defined the problem and devel- 
oped a better understanding of what factors might be contributing to the problem. The team agreed that student 
discourse in math lessons was an area in which students lacked skills and noted that some students were much 
less able to engage in student discourse than others. They agreed that this was a problem they could address that 
might lead to improvement in overall student engagement and achievement in math. 
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Meeting 1 


Goals 


e Introduce Improvement Team members to continuous improvement as a set of processes that are relevant to 


school- and district-based teams. 


e Discuss and clarify the focus for shared work. 


e Discuss next steps in implementing the continuous improvement effort and an overall plan for the year. 


Agenda (90 minutes) 


Topic 


Welcome, review of agenda and 
goals, introductions, and discussion 
of pre-meeting reading 


Watch and discuss Video 1 


Activity: Read continuous 
improvement scenarios and discuss 
guiding questions 


Activity: Discuss the overall focus 


Improvement Team member roles 
and responsibilities and calendar 


Wrap up and next steps 


\Weelaclarel is 


Toolkit, including a customized agenda; pre-meeting reading 
(p. II-3) 


Video 1 (https://youtu.be/wr-7S8KRKGE) 


Handout 7—Continuous improvement scenarios and guiding 
questions (p. II-9) 


Toolkit, Handout 2—Preliminary data inventory worksheet 

(p. I-8) 

Handout 4—Improvement Team member roles and responsibilities 
(p. 1-13), Handout 5—Continuous improvement calendar 

(p. I-16 and https://ies.ed.gov/ncee/edlabs/regions/northeast/ 
pdf/handout5_Cl-calendar.pdf) 


Toolkit 


Time 


15 minutes 


15 minutes 


20 minutes 


25 minutes 


10 minutes 


5 minutes 


Welcome, review of agenda and goals, introductions, and discussion of pre-meeting reading (15 minutes) 


Begin the meeting with a welcome, a review of the agenda and goals, introductions, and a short discussion of the 
pre-meeting reading. Improvement Team members should: 


e Introduce themselves. 


e Describe one thing they would like to improve outside of work. Examples: drink more water, spend more time 
reading for pleasure, get out of the house in the morning in less of a rush. 


e Present one question they have about continuous improvement, including PDSA cycles, based on the pre-meet- 


ing reading. 


The meeting facilitator or the recorder should take notes on the questions the team members raise. 


Watch and discuss Video 1 (15 minutes) 


Video 1 (available at https://youtu.be/wr-7S8KRKGE) tells the story of a group of teachers and their principal at 
Grove Elementary School who used continuous improvement to address a problem that they saw with their stu- 
dents’ math learning. The video provides a general overview of their work and their reflections on the process. 
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After watching the video, discuss how the teachers and principals described the continuous improvement effort, 
what challenges they faced, and what they believe they achieved. 


Activity: Read continuous improvement scenarios and discuss guiding questions (20 minutes) 


NKolKom Kolm aale\eiulateami-lelliv-lnels 
Depending on the number of Improvement Team members, assign pairs or triads 


to review one scenario each and provide an overview of that scenario for the larger 
group. The guiding questions at the end of each scenario may be useful in prompting 
discussion. 


Read Handout 7—Continuous improvement scenarios and guiding questions (see next page) and discuss the sce- 
narios and guiding questions. The scenarios are meant to illustrate the different types of issues or problems that 
are appropriate to address through continuous improvement. Review the cases to understand the range of prob- 
lems for which continuous improvement can be a useful approach, paying particular attention to the problems 
embedded in the scenarios and how the educators might go about clarifying the problem, engaging key stake- 
holders, and examining relevant data. 
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Handout 7. Continuous improvement scenarios and guiding 
questions 


Scenario 1: Darlene and Response to Intervention progress monitoring 


Darlene is the digital learning coordinator in her school, where she has worked for several years. One of her 
responsibilities is overseeing the use of the school’s Response to Intervention (RTI) online progress monitoring 
reading assessment. The teachers administer the assessment according to the school’s 30-day instructional cycle 
schedule. The assessment was in use at the school before Darlene arrived, and all the teachers have successfully 
administered and used it. Teachers make time in their schedules to administer the assessment and are good at 
following up with students who were absent to ensure that they take the assessment as soon as possible. And the 
teachers talk about the assessment results in their grade team meetings when Darlene drops in. 


Given the success of the reading assessment for progress monitoring, this year the principal decided, with faculty 
support, to start using the same company’s math assessment for a similar 30-day RT| progress monitoring for math. 


Darlene could not have been more surprised by the rollout of the math assessment: teachers weren’t administering 
it reliably or reviewing the data reports from it. And when Darlene observed classrooms, it seemed that students 
were not very engaged with the assessment and were just rushing through it, which did not seem to be the case 
with the reading assessment. The company designed the reading and math assessments to be similar: the admin- 
istration time was the same, the data reports were the same, and the look and feel to the students was the same. 


So why was the reading assessment such a success while the math assessment was not? Darlene talked to a few 
teachers in one-on-one settings about this, and she was surprised to learn that teachers did not understand the 
math-related data reports as well as Darlene had thought they did. Some teachers did not even know that many 
of these data reports existed. Because the teachers did not use the data reports, they did not see the value of the 
math assessment. Darlene guessed that students picked up on this perception and that it contributed to what she 
observed—that students were not taking the math assessment seriously. 


To address this issue, Darlene asked the principal for her support in starting an Improvement Team with the 
grade-level leaders in her school. Darlene wants to use the continuous improvement process to improve the facul- 
ty’s assessment literacy and understanding of how the reading and math assessment data reports can help teach- 
ers create instructional groupings of students who need intervention and corrective instruction. Darlene believes 
that if teachers could see their struggling students accelerate their learning and move toward grade-level achieve- 
ment using the data reports, the teachers would see the value of administering both assessments as intended. 
Guiding questions 

e What is the overall issue that Darlene wants to address? 

e Who should be part of her Improvement Team? 

e¢ How should she present the issue to the Improvement Team? 

e What questions might she ask them in seeking further input on the issue or problem? 


e What data might she collect to help the Improvement Team understand the issue? 
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Scenario 2: Richard and instructional practices 


Richard recently became the principal of Super Star High School. When he arrived, the school was struggling in 
several areas, and teacher morale was low, according to numerous school and district staff members. In addition, 
through classroom walkthroughs he observed that many teachers were not using effective classroom manage- 
ment strategies, which he concluded could be contributing to low teacher morale and to high rates of student 
discipline problems. He was also concerned that valuable instructional minutes were being lost. 


At the middle school where he had been an assistant principal, Richard and a team of instructional coaches had 
successfully employed frequent classroom observation to support the implementation of several evidence-based 
instructional practices. The practices included a bell-to-bell instructional framework to maximize instructional 
minutes, improved classroom management to reduce disruptions, and more efficient lesson planning to reduce 
transition time between activities. Instruction was improved, school climate improved, and students received 
more instruction per day and throughout the year. Given his past success with leading an instructional coaching 
team to change instructional practices in these areas, Richard set out to re-create his past success at Super Star 
High School. 


Unfortunately, Richard’s attempts to implement frequent and targeted observations and feedback to support 
evidence-based practices at Super Star High School were met with resistance from several teachers. The teach- 
ers at Super Star High School were not used to having instructional coaches and administrators in their class- 
rooms on a regular basis. Most teachers were observed and provided with feedback for their formal evaluations 
only as part of a required but far from meaningful activity. Only the least successful teachers received additional 
ongoing attention throughout the year, which resulted in many teachers reacting to the increased engagement 
from instructional coaches as a signal that they were now among the lowest performing teachers in the school. 
Richard could tell he had his work cut out for him. It wasn’t going to be easy to change the professional culture 
and the instructional practices of Super Star High School teachers. He wanted to find a way to effectively build 
collaborative and supportive working relationships while simultaneously creating a sense of urgency for improv- 
ing classroom management, use of instructional minutes, and school discipline. 

Guiding questions 

e What is the overall issue that Richard wants to address? 

e Who should be part of his Improvement Team? 

e How should he present the issue to the Improvement Team? 


e What questions might he ask them in seeking further input on the issue or problem? 


e What data might he collect to help the Improvement Team understand the issue? 
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Scenario 3: Cyndi and Next Generation Science Standards implementation 


Cyndi’s elementary school has adopted a new science curriculum for upper elementary grades that is aligned 
with the Next Generation Science Standards (NGSS). As the grade 4 teacher leader, Cyndi is excited about this 
opportunity to use a new science curriculum with her students. NGSS is focused on student sense-making using 
direct experience of science concepts, and Cyndi loves that the new curriculum comes with a variety of investiga- 
tions, material kits, experiments, and simulations for students to work on in groups to get hands-on experience 
with science. She also appreciates that the curriculum is aligned with Common Core English Language Arts Stan- 
dards and comes with recommendations and strategies for making reading and writing about science accessible 
for students. The curriculum includes a system for organizing students’ science notebooks with printouts and 
pre-created tables, which Cyndi thinks will reduce the amount of time teachers spend helping students orga- 
nize their science work and enable teachers to focus more time on science learning, science writing, and science 
discussions. 


However, with all the opportunities for new teaching methods and new activities comes a lot of teacher anxiety. 
Cyndi is concerned that some of the grade 4 teachers will be anxious about implementing the new curriculum. The 
curriculum requires teachers to shift from teacher-led activities to more student-led activities, and many teach- 
ers are concerned about classroom management and off-task behavior. The curriculum also includes a strand of 
earth science content that some teachers have never had to teach before. The last major concern is the amount 
of science writing that students are expected to do. Although the grade 4 teaching team unanimously agrees that 
writing in all subjects is important for developing students’ literacy and critical thinking skills, quite a few teachers 
feel intimidated by the volume of writing that students are expected to put in their science notebooks and that 
teachers will be expected to review to support their students’ growth. 


As is usual for Cyndi, she has a hundred ideas about things she can do to support her teachers in implementing 
this curriculum. But she is not sure which ideas are going to resonate with her teachers, nor is she sure what 
support teachers feel they most need. 

Guiding questions 

e What is the overall issue that Cyndi wants to address? 

e Who should be part of her Improvement Team? 

¢ How should she present the issue to the Improvement Team? 


e What questions might she ask them in seeking further input on the issue or problem? 


e What data might she collect to help the Improvement Team to understand the issue? 
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Activity: Discuss the overall focus (25 minutes) 


NKolKom Kol aal=\eiulatemr-lellir-leels 


It is valuable to frame the discussion of the overall focus with some general guidance about what 

the Improvement Team should achieve during this initial meeting: 

e One of the primary objectives of this first formal meeting of the Improvement Team is to build 
a shared understanding of the overall focus that the Improvement Team gathered to address. 

e Continuous improvement is an inherently collaborative process in which a team works 
together to build a shared understanding, set a shared direction, and build a shared action 
plan with shared responsibilities. 

e In setting the stage for the Improvement Team’s work together, it is important to recognize 
that members may have different experiences and perspectives related to the identified 
focus. This may lead to different ideas about what the problem is and how it should be solved. 
This is in fact a strength of the continuous improvement approach, because these multiple 
perspectives contribute to a rich and varied set of possible approaches that can be used to 
solve the common problem of practice. 


Discuss and further clarify the overall focus for the continuous improvement effort using the four steps below. 
At this point the team needs to agree only on the overall focus of its work and not on the specific problem and 
change practices to implement. That will come in a future meeting. The objectives of this discussion are to: 


e Share multiple perspectives and draw people’s attention to the overall focus. 
e Identify additional questions that Improvement Team members have about the overall focus. 


e Determine whether and what data or information collection related to the overall focus will occur before the 
next meeting. 


Step 1. Have the Cl leader or meeting facilitator present a brief statement about the overall focus that the team 
will address through the continuous improvement effort. For the Grove Elementary School example case the 
statement is: 


Student achievement in elementary math is an area of concern, based on recent state achievement results. 
Step 2. Discuss the overall focus, answering the following questions: 


e What issues or challenges do we face in our own context as a classroom teacher, coach, curriculum coordina- 
tor, and so forth that are related to this focus? (For the Grove Elementary School example case, teachers would 
describe their difficulty getting students to engage in class or to do more complex multistep problems.) 


e What successes have we had related to this focus? What has gone well and why? (For the Grove Elementary 
School example case, teachers would describe their success using small groups or having students complete “do 
now’s” of short review problems before they complete a math lesson.) 


Step 3. Brainstorm questions about the overall focus and what data related to the issue would be valuable to 
collect or review: 


e What questions do we have about why we are facing these challenges and why we are experiencing these 
successes? (For the Grove Elementary School example case, teachers would want to know more about which 
students are engaged in math class and which are not and whether there are certain types of “do now” review 
problems that students have difficulty completing.) 
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e What data might help us answer these questions? (For the Grove Elementary School example case, teachers 
would consider tallying which students appeared engaged in math class or collecting and reviewing the “do 
now” review questions to determine which types of problems were more difficult for students.) 


Step 4. Have the Cl leader share Handout 2—Preliminary data inventory worksheet (p. |-8) that he or she com- 
pleted to help inform which data should be collected. For data related to the questions that arose in the discus- 
sion, determine what needs to be collected and who will collect it before Meeting 2. 


The meeting facilitator or designated recorder should take notes on this discussion. 
Improvement Team member roles and responsibilities and calendar (10 minutes) 


To conclude the meeting, review Handout 4—Improvement Team member roles and responsibilities (p. |-13) 
and determine who will take on what responsibilities (if this has already been accomplished, have the Cl leader 
present these roles), and review Handout 5—Continuous improvement calendar (p. |-16 and https://ies.ed.gov/ 
ncee/edlabs/regions/northeast/pdf/handout5_Cl-calendar.pdf) and discuss the dates and times for upcoming 
meetings, as well as the overall timeline for the work. 


Wrap up and next steps (5 minutes) 
Cl leader 


e Send the notes from the Meeting 1 to Improvement Team members and ask them to review the notes before 
Meeting 2. (If a recorder has been selected, he or she can do this task.) 


e Ask Improvement Team members to complete the pre-meeting reading for Meeting 2. 


e Collect any data or information related to the overall focus discussed in the meeting and prepare to share it in 
Meeting 2. This may include working with the data manager or other Improvement Team members. 


e Prepare flip chart paper or a shared screen that can be used to record ideas generated in Meeting 2. 


Assign a team member to facilitate Meeting 2, as needed. 
Improvement Team members 


e Read the notes from Meeting 1 and complete the pre-meeting reading for Meeting 2 on defining the problem 
and developing a fishbone diagram. 


e Collect any data or information related to the overall focus discussed in the meeting and prepare to share it in 
Meeting 2. 


Data manager 


e Work with the Cl leader and other Improvement Team members to collect any data or information related to 
the overall focus discussed in the meeting and prepare to share it in Meeting 2. 


Recorder 


e Send the notes from Meeting 1 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for Meeting 2. 
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Meeting 2. Defining the problem and determining root 
causes using a fishbone diagram 


In Meeting 1 the Improvement Team discussed and reached some con- Materials needed 

sensus about the overarching issue that the team will address. In this Pre-meeting reading: Defining the 
meeting the team will learn about the “fishbone” or “cause and effect” problem and developing a fishbone 
diagram, which is a tool to support a team in clearly defining a problem diagram (p. II-15) 

of practice and what might be contributing to the problem. The team Handout 8—Fishbone diagram 
will then develop a fishbone diagram related to the defined problem for the Grove Elementary School 
of practice. This will set the team up to develop an aim statement and example case (p. IIl-17 and 


identify change practices in Meeting 3. https://ies.ed.gov/ncee/edlabs/ 
regions/northeast/pdf/handout8 __ 

Preparations Fishbone-example.pdf) 

Handout 9—Template for a 


Cl leader fishbone diagram (p. II-22 and 


e Ask Improvement Team members to review the notes from Meeting 1. https://ies.ed.gov/ncee/edlabs/ 


regions/northeast/pdf/handout9_ 


e Ask Improvement Team members to complete the pre-meeting reading Fishbone-template.pdf) 
(see next page) on defining the problem and developing a fishbone Handout 10—Guiding questions 
diagram and to review Handout 8—Fishbone diagram for the Grove for developing a fishbone diagram 


Elementary School example case (p. |I-17 and https://ies.ed.gov/ncee/ (p. 11-23) 
edlabs/regions/northeast/pdf/handout8_Fishbone-example.pdf). 


e If any data or information was collected related to the overall focus discussed in Meeting 1, prepare to share it 
in the meeting. 


e Prepare flip chart paper or a shared screen that can be used to record ideas generated during the defining the 
problem activity. 


e Assign a team member to facilitate the meeting, as needed. 
Improvement Team members 


e Read the notes from Meeting 1, complete the pre-meeting reading (see next page) on defining the problem and 
developing a fishbone diagram, and review Handout 8—Fishbone diagram for the Grove Elementary School example 
case (p. Il-17 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout8_Fishbone-example.pdf). 


e Bring any data or information that was collected related to the overall focus discussed in Meeting 1, and 
prepare to share it in the meeting. 


¢ Come prepared to define the problem of practice. 
Data manager 


e Work with the Cl leader and other Improvement Team members to prepare any data or information that was 
collected related to the overall focus discussed in Meeting 1. 


Recorder 


e Send the notes from Meeting 1 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for the meeting. 1-14 


Pre-meeting reading: Defining the problem and developing a 
fishbone diagram 


Grove Elementary School example case: Part II 


In Meeting 1 you were introduced to the Grove Elementary School example case and heard from some practitioners 
in Video 1. Meeting 2 begins with an activity related to this example case. Below, the example case is provided 
again but with some additional information about defining the problem and developing a fishbone diagram. As 
you read, consider the following questions: What was the problem that the team identified? What practices did the 
team consider to address the problem it had identified? What resources or supports might be needed to implement 
the selected change practice? 


Grove Elementary School is in the second year of implementing a new curriculum and instructional strategies that 
are aligned with the Common Core Standards for Mathematical Practice, which define a range of skills that math 
educators should seek to develop in their students. The school recently received its state assessment data, and 
administrators noticed that math scores were lower than in prior years, particularly among low-income students. 
The principal has also observed that some teachers lack sufficient content or pedagogical knowledge to provide 
students with instruction that matches the Common Core Standards; she has observed this specifically in engag- 
ing students in math discourse. 


The principal met with a group of teachers to discuss the lower math scores and the challenges associated with 
implementing the new curriculum and standards. The group discussed the standards for math practice and things 
that were and were not working in the classroom. The teachers agreed that what was working well is that stu- 
dents were displaying quantitative reasoning and could use a variety of materials to practice their skills. However, 
teachers were concerned that many of their students could neither construct viable arguments nor analyze sit- 
uations and communicate them to others. To better understand teachers’ concerns, the principal decided to 
observe some teachers during math lessons. 


The principal selected teachers at different grade levels to observe and used a simple protocol to track teacher 
questions and student responses. She noted that some students seemed to be less engaged during the lessons. 
The principal met again with the teachers and shared the results of the initial observations, focusing on what the 
teachers were doing and on the nature and frequency of student responses. During this meeting, after reviewing 
the results of the principal’s data collection, the Improvement Team more clearly defined the problem and devel- 
oped a better understanding of what factors might be contributing to the problem through a fishbone diagram. 
The team agreed that student discourse in math lessons was an area in which students lacked skills and noted 
that some students were much less able to engage in discourse than others. They agreed that this was a problem 
they could address that might lead to improvement in overall student engagement and achievement in math. 


After developing the fishbone diagram in Meeting 2 (see below), the Improvement Team turned from the problem 
to potential solutions through a driver diagram in Meeting 3. Since the team agreed that increasing student 
engagement by promoting higher level thinking and math discourse was a problem it could address, it concluded 
that it was necessary to build teachers’ capacity to promote the math discourse of all learners. The teachers con- 
sidered various practices to test—including implementing small group activities, employing team teaching, and 
conducting mini-lessons—and ultimately decided on a set of “sentence starters and frames” to promote high- 
level math discourse in their classrooms. These sentence starters and frames are simple phrases that students 
can use to explain and justify their techniques for solving math problems. The sentence starters also support 
conceptual understanding as the teacher facilitates discussions and invites students to ask open-ended questions 
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that encourage critical thinking. The teachers selected this change practice over the others because they believed 
it represented a concrete change from their current practice, because it was something they could all implement 
fairly similarly, and because they could collect some basic data during their instructional time that would allow 
them to measure an increase in student math discourse. 


Developing a fishbone diagram 


Before testing any change practices, the Improvement Team needs to define the problem to be solved and identi- 
fy potential strategies or changes that might be expected to achieve the aim of the project. A fishbone diagram is 
a graphic representation of everything that might contribute to a problem. Building a fishbone diagram supports 
the team in drawing on their diverse perspectives to develop a clearer picture of the problem and potential ways 
to address it. Generating all the factors that might contribute to the problem before homing in on a proposed 
solution also ensures that participants consider all aspects of the issue before they define the practice or process 
to be tested. A fishbone diagram contains three main parts: 


¢ Problem statement. A brief statement of the specific and measurable problem that the Improvement Team will 
address. 


e Factors. |\deas regarding the issues that contribute to the problem. In the example, one factor is that students 
are not engaged in their learning, which the Improvement Team believes has led to the problem that students 
are below proficient in math. 


° Causes. All the possible reasons related to each of the factors. In the example, it would not be sufficient to just 
state that students are not engaged. The team must dig deeper to try to uncover what is contributing to, or 
causing, the lack of engagement. For example, the team posits that one of several causes of students’ limited 
engagement is that they lack language and listening skills. 
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Handout 8. Fishbone diagram for the Grove Elementary 


School example case 


:REL Fishbone diagram 


& ISLANDS 


Regional Edvcsonal Laboratory 
‘MEDC 


This completed fishbone diagram shows the 
problem that needs to be solved, factors that 


contribute to the problem, and the potential causes Students are not engaged 
of each factor. The main problem is located on the in their learning 
far left with the potential causes cascading from erica tekdeen 
the “major bones” (factors) of the diagram. i 
Participation in class activities understanding of math 
is not equitable across students concepts 


Students lack language and 
listening skills 


© Causes 


Students don’t see the relevance Students have difficulty 


of math in their lives 


Students lack confidence 
in math 


Some students just look to 
get the “right answer” 


© Problem Statement 


Students in elementary grades 
are below proficient in math and 
low-income students are 
disproportionately represented 
among those students. 


Teachers have received limited 
professional development on 

math instruction that promotes 
engagement 


The curriculum lacks open, 
authentic tasks 


Teachers have limited access to 
tools and models for developing 
instructional activities 


Math activities are too 
procedurally focused 


Teachers are new to the 
school’s math curriculum 


e Factors Curriculum is not rigorous 


Teachers need additional 
supports to provide effective 
math instruction 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 


Students are more focused 
on procedures than concepts 


visualizing math concepts 
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Meeting 2 
Goal 


e Collaboratively define the problem of practice by developing a fishbone diagram for the focus identified in 
Meeting 1. 


Agenda (90 minutes) 


Materials 
Welcome and review of agenda and goals Meeting 1 notes, agenda 10 minutes 
Introduce the pre-meeting reading and Pre-meeting reading (p. II-15), Model for Improvement 15 minutes 
activity chart (p. I-19) 
Watch Video 2 and discuss Handout 8— Video 2 (https://youtu.be/IfVEJVWR3al), Handout 8— 15 minutes 
Fishbone diagram for the Grove Elementary Fishbone diagram for the Grove Elementary School 
School example case example case (p. |I-17 and https://ies.ed.gov/ncee/edlabs/ 
regions/northeast/pdf/handout8_Fishbone-example.pdf) 
Activity: Develop a fishbone diagram Handout 9—Template for a fishbone diagram (p. \|-22 and | 40 minutes 


https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/ 
handout9_Fishbone-template.pdf), flip chart, sticky notes, 
notes from Meeting 1, any data or information collected, 
Handout 10—Guiding questions for developing a fishbone 
diagram (p. II-23) 


Wrap up and next steps 10 minutes 


Welcome and review of agenda and goals (10 minutes) 


Begin the meeting with a welcome and a review of the agenda and goals. The meeting will center on developing a 
fishbone diagram related to the overall focus discussed in Meeting 1. 


Take a few minutes to have Improvement Team members: 
e Ask any questions about the last meeting. 


e Indicate whether they have data or information related to the overall focus to share at this meeting. This will 
be useful as part of the discussion of the fishbone diagram. 
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Introduce the pre-meeting reading and activity (15 minutes) 


NKolKom Ko aal=\eiulatemi-lellir-leels 


The Improvement Team members should have read the Grove Elementary School 
example case, provided in the pre-meeting reading, before the meeting. If they 
haven't, first have them read the case. The activity that follows is designed to help 
them think about how to define a specific problem. 


Using the pre-meeting reading on the Grove Elementary School example case, discuss and answer the questions 
below in the Model for Improvement chart. Record notes for each question in the appropriate column. 


What is the specific problem What changes do the teachers How will the teachers at Grove 


that the teachers at Grove choose to introduce to Elementary know that their change 
Elementary are trying to solve? address the problem? is an actual improvement? 


Additional questions to discuss: 


¢ What do you notice about the size or scope (the “grain size”) of the problem that the principal and teachers are 
trying to solve? 


e What do you notice about the proposed change practice? Is it likely to achieve the goals they want to achieve? 
Is it specific? Is it actionable? 


Watch Video 2 and discuss Handout 8—Fishbone diagram for the Grove Elementary School example case (15 minutes) 


Video 2 (available at https://youtu.be/If VEJVWR3al) explains what a fishbone diagram is and uses the Grove Ele- 
mentary School example case to illustrate how to build one. 


After watching the video, review Handout 8—Fishbone diagram for the Grove Elementary School example case 
(p. II-17 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout8_Fishbone-example.pdf) and discuss 


the following three questions: 


e What is the relationship between the factors and the problem identified in the Grove Elementary School 
example case? 


e Are all the factors, and related causes, within the control or purview of the teachers and principal to address? 
¢ Does the fishbone diagram make sense in light of what you know about the case? 


11-19 


Activity: Develop a fishbone diagram (40 minutes) 
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Using a flip chart and sticky notes, Improvement Team members could brainstorm 
factors and causes of those factors and write each idea on a separate sticky note and 
then post these notes to a blank fishbone diagram. This will allow the team to move 
the ideas around in real time and to discuss what causes are related to which factors. 


Now it is the Improvement Team’s turn to develop a fishbone 
diagram for the focus generated in Meeting 1. Follow the five steps 
below to develop a fishbone diagram using a flip chart and sticky 
notes or Handout 9—Template for a fishbone diagram (p. Il-22 and 
https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout9 __ 
Fishbone-template.pdf). Refer to Handout 10—Guiding questions 
for developing a fishbone diagram (p. \I-23) while developing the 
diagram. Though 40 minutes is the suggested timeframe, the team 
may need more time for this activity. 


Step 1. Review the notes from Meeting 1 about issues and prob- 
lems the Improvement Team faces related to the overall focus 
and promising practices or approaches that have been witnessed 
or tried. The team should agree generally on the overall focus but 
does not need to totally agree on why they are seeing the issue 
or issues they have identified. Remember that the purpose of the 
fishbone diagram is to determine why a specific problem occurs 
and factors that may contribute to it. There may be, and probably 
are, multiple contributing factors. 


Step 2. As a result of the discussion at the last meeting, some 
Improvement Team members may have collected data or infor- 
mation to support the defining the problem activity. Review these 
data and discuss what they might contribute to the understanding 
of the problem. See the Preliminary data review sidebar for a brief 
description of how to review and discuss these data. 


Preliminary data review 


Identifying and collecting preliminary data 

prior to the first Plan-Do-Study-Act (PDSA) 

cycle may help the Improvement Team better 
understand and define the problem. During 

Meeting 1 the team may have identified some 

preliminary data related to the overall issue. 

These data should be included in the discus- 

sion when developing the fishbone diagram. 

Here are some general principles for examin- 

ing these data: 

e Describe the data before interpreting or 
making conclusions based on the data: 
What are the data? 

e¢ Observe any patterns or trends: What do 
you notice about the data overall? 

¢ Note how the data relate to the problem: 
Do the data reinforce or contradict what 
you see as the issue or problem you would 
like to address? 

e Determine how the data will inform your 
problem definition: How will the data 
help further specify the problem you will 
address? 


Step 3. Based on the initial discussion and any data review, develop a draft statement of the problem. This one- 
sentence summation of the problem of practice is placed on the left side of the fishbone diagram. In the Grove 
Elementary School example case, one of the main problems that the teachers and principal identified was that 
students, especially low-income students, were below proficient in math. That is the problem that the Improve- 


ment Team at Grove Elementary School chose to focus on. 
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Step 4. Generate the “major bones” in the diagram. Each major bone represents a main factor that may be related 
to the problem. For example, a lack of student engagement is one major factor the Improvement Team at Grove 
Elementary School identified. 


Step 5. Within each major bone, brainstorm more specific causes related to each key factor. A fishbone diagram 
will typically include four to six major bones (main factors), with a few causes cascading from each factor. 


Brainstorming and organizing factors that contribute to the problem enable the Improvement Team to focus on 
why the problem occurs, not just on symptoms of the problem. The main idea behind a fishbone diagram is to 
brainstorm all possible factors and then drill down to identify what might be causing the problem. Data play a 
critical role in defining the problem and developing the fishbone diagram, but the team members should not limit 
causes to those that are directly related to data they collect or to which they have access. The value of the team’s 
multiple perspectives is the rich discussion about the issues faced. Once team members exhaust their ideas about 
why the problem might exist, they can begin to plan ways to address the problem. 
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Handout 9. Template for a fishbone diagram 


La. 5 
vomest Fishbone diagram 


To complete your own fishbone diagram, begin 
with the problem statement on the far left. Then 
identify factors that contribute to the problem (blue 
boxes) and likely causes for those factors. 


© Causes 


© Problem Statement 


© Factors 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 
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Handout 10. Guiding questions for developing a fishbone 
diagram 


Consider the guiding questions below while developing and reviewing the fishbone diagram. Remember that the 
fishbone diagram helps in determining and clarifying the problem. The questions below are designed to prompt 
the team to consider the problem from many angles. It is from this examination of the many factors, and causes 
of those factors embedded within the problem, that some of the best ideas about how to address the problem 
will emerge. 

Problem statement 

e What problem are you trying to address? 

e Is the problem clear, specific, and measurable? 

e Is the problem clearly articulated? 

¢ Does the problem seem too big for the Improvement Team to address? Is the problem too general? Is it unre- 
alistic to think the team could make any changes related to the problem? If the answer is yes to any of these 
questions, revisit the problem and try to focus or narrow it. 

Factors 

e Are the factors contributing to the problem? Are they directly related to the problem? 

e Are the factors just symptoms of the problem? In other words, are they evidence of the problem rather than 
actual conditions or circumstances that contribute to the problem? If the answer is yes to any of these ques- 
tions, continue to brainstorm factors that are causing the problem. 

¢ Do the factors represent the multiple perspectives of the Improvement Team? 

e Is any critical perspective missing? 

Causes 

e Are the potential causes directly related to the factors? 


e Are the potential causes clearly stated? 


e Are the causes at a more specific grain size than the factors? In other words, do they go further than the 
factors in terms of specifying the problem? 
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Wrap up and next steps (10 minutes) 

Based on the conclusion of the fishbone diagram activity, the final step of the meeting will be to confirm that 

the work generated will be compiled and that a draft fishbone diagram will be developed and shared with the 

Improvement Team before the next meeting. Determine how this will be completed and who will do it. 

Cl leader 

¢ Complete a draft fishbone diagram based on the work generated during the meeting, using Handout 9—Template 
for a fishbone diagram (p. \I-22 and _https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout9_ 
Fishbone-template.pdf) and share it with team members before Meeting 3. Include Handout 10— Guiding ques- 
tions for developing a fishbone diagram (p. I-23) so team members can reflect on and review the fishbone 
diagram at the start of Meeting 3. (The Cl leader can designate another team member to complete this task.) 

e Ask Improvement Team members to complete the pre-meeting reading for Meeting 3, which describes a driver 
diagram, and to review Handout 11—Driver diagram for the Grove Elementary School example case (p. \I-28 
and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout11_Driver-diagram-example.pdf). 

Improvement Team members 

e Review the draft fishbone diagram and come to Meeting 3 with any questions or suggestions for changes. 

¢ Complete the pre-meeting reading for Meeting 3, which describes a driver diagram and review Handout 11— 
Driver diagram for the Grove Elementary School example case (p. II-28 and https://ies.ed.gov/ncee/edlabs/ 
regions/northeast/pdf/handout11_Driver-diagram-example.pdf). 

Data manager 

e None. 


Recorder 


e Send the notes from Meeting 2 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for Meeting 3. 
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Meeting 3. Establishing an aim and developing a driver diagram 


The purpose of this meeting is to arrive at an aim for the continuous improvement effort and to develop a driver 


diagram. As a reminder, the aim is the specific goal, developed in 
response to the problem statement, that guides the improvement effort. 
It should describe what the Improvement Team wants to achieve and 
should be specific and measurable. A driver diagram is essential because 
the specific change practices that the Improvement Team will implement 
are derived from it. A driver diagram also serves as the basis for Improve- 
ment Team members’ determination of the appropriate data to collect in 
efforts to understand how the change practices are working and whether 
they are leading to improvement. 


Preparations 
Cl leader 


¢ Complete a draft fishbone diagram based on the work generated 
during Meeting 2, using Handout 9—Template for a fishbone diagram 
(p. Il-22 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/ 
handout9_Fishbone-template.pdf), share it with the Improvement 
Team before the meeting, and be prepared to discuss it with team 
members using Handout 10—Guiding questions for developing a fish- 
bone diagram (p. II-23) for guidance. 


e Ask Improvement Team members to complete the pre-meeting 
reading (see next page), which describes a driver diagram, and to 
review Handout 11—Driver diagram for the Grove Elementary School 
example case (p. I-28 and https://ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout11_Driver-diagram-example.pdf). 


Improvement Team members 


Materials needed 


Draft fishbone diagram developed 
in Meeting 2 

Pre-meeting reading: Description of 
a driver diagram (p. Il-26) 

Handout 8—Fishbone diagram 

for the Grove Elementary School 


example case (p. II-17 and 
https://ies.ed.gov/ncee/edlabs/ 
regions/northeast/pdf/handout8 __ 


Fishbone-example.pdf) 

Handout 10—Guiding questions 
for developing a fishbone diagram 
(p. 1-23) 

Handout 11—Driver diagram for 
the Grove Elementary School 
example case (p. II-28 and https:// 
ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout11_Driver- 
diagram-example.pdf) 

Handout 12—Template for a driver 
diagram (p. II-32 and https:// 
ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout12_Driver- 
diagram-template.pdf) 

Handout 13— Guiding questions 
for developing a driver diagram 
(p. 1-33) 


e Review the draft fishbone diagram developed in Meeting 2 and come prepared to discuss and make any needed 
changes using Handout 10—Guiding questions for developing a fishbone diagram (p. ||-23) for guidance. 


¢ Complete the pre-meeting reading (see next page), which describes a driver diagram, and review Handout 11— 
Driver diagram for the Grove Elementary School example case (p. II-28 and https://ies.ed.gov/ncee/edlabs/ 


regions/northeast/pdf/handout11_Driver-diagram-example.pdf). 
Data manager 
e None. 


Recorder 


e Send the notes from Meeting 2 in a timely manner to all Improvement Team members and save the notes ina 


shared location that is accessible to all team members. 


e Prepare to take notes for the meeting. 
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Pre-meeting reading: Description of a driver diagram 


A driver diagram is a tool that helps translate the work from the fishbone diagram—which defined the problem, 
main factors, and related causes—into a clearly articulated improvement goal, or aim. The driver diagram identi- 
fies a logical set of smaller, tangible goals and supports the selection of specific actions, or change practices, to be 
tested as part of the continuous improvement process. The driver diagram provides a graphic representation of 
the various actions and change practices the Improvement Team could test through the continuous improvement 
effort. It has up to four elements: an aim statement, primary drivers, secondary drivers, and change practices. 


Aim statement. The aim statement is the specific goal, developed in response to the problem statement, that 
guides the improvement effort. It should describe what the Improvement Team wants to achieve, by how much 
the team wants to increase or decrease something, and, if possible or appropriate, by when. Ideally, an aim 
statement should target a specific population, be time specific, and be measurable. The aim statement generally 
restates the problem from the fishbone diagram in the affirmative. In the Grove Elementary School example case 
the problem is that students, especially low-income students, are below proficient in math. The aim statement 
restates this problem by affirming what the positive corollary to that problem might be and states it in specific 
and measurable terms. The aim statement in the driver diagram for the example case is that the percentage of 
students performing at proficient or above will increase by 10 percent, and the gap between low-income students 
and their peers will decrease. 


Primary drivers. Primary drivers focus on the conditions that are essential for making the improvement described 
in the aim statement. An aim generally has three to five primary drivers that act independently or together. These 
drivers are derived from the factors identified in the fishbone diagram. Taken together, the primary drivers repre- 
sent how the aim might be achieved. In the Grove Elementary School example case the principal and the teachers 
observed that students were not engaged in math class, which is identified as a factor in the driver diagram. They 
believe this lack of engagement might be contributing to lower performance. So, one of the primary drivers they 
identified is to increase student engagement in class. However, the primary drivers may be too general, as with 
this example, to direct specific actions, in which case, secondary drivers are necessary. 


Secondary drivers. Secondary drivers, derived from the primary drivers, are specific leverage points that are 
expected to have a direct impact on the primary drivers and, in turn, on the aim. They more clearly direct the 
types of change practices that participants can implement. In the Grove Elementary School example case a 
primary driver, related to one of the factors in the fishbone diagram, is to increase student engagement. So, a sec- 
ondary driver that directly relates to student engagement is to increase students’ math discourse. This secondary 
driver is derived from one of the causes the team identifies: a lack of language and listening skills. While this focus 
on student discourse is somewhat more specific, and is expected to have an impact on the primary driver, it still 
does not provide a specific action in the form of a change practice. Depending on the scope of the aim and the 
specificity of the primary drivers, secondary drivers may not be necessary. 


Change practices. Change practices are the interventions or specific work practices that are predicted to affect 
the secondary and, in turn, the primary drivers. Derived from the secondary drivers, or in some cases from the 
primary drivers, change practices should be specific, actionable, and measurable. In the Grove Elementary School 
example case a change practice directly related to the secondary driver of increasing math discourse among stu- 
dents is to have teachers introduce and use sentence starters to prompt math discourse in the classroom. This 
change practice is specific and measurable and is hypothesized to positively influence math discourse and, in 
turn, student engagement. 
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As the Improvement Team translates the primary drivers into secondary drivers and then into change practices, 
the specific tools, processes, and strategies to be used should become clear. Through this process the team can 
clarify all the possible options and then select the best, most manageable approach. Handout 11—Driver diagram 
for the Grove Elementary School example case (see next page) shows that the Grove Elementary School Improve- 
ment Team identified several potential change practices, but they settled on sentence starters as the change 
practice to implement because it: 


e Represented a concrete change from their current practice. 
e Was something that could be implemented fairly similarly acros all classrooms. 


e Was a practice about which all teachers could collect some basic data during their instructional time that would 
allow them to measure an increase in student math discourse. 


Driver diagrams should be living documents that change over time as Improvement Team members see changes 
in their setting. The diagrams represent the team’s theory of change—or idea about what strategies are most 
likely to achieve the goal outlined in the aim statement—but they are always provisional and, through the con- 
tinuous improvement effort, are tested and revised. Since the driver diagram is designed to be an improvement 
tool, the team should select drivers that are realistic and most likely to affect the aim and should remove drivers 
that are likely to have little to no impact or that have no chance of being influenced. In the Grove Elementary 
School example case the primary and secondary drivers are meant to affect the aim and be realistic, specific, and 
achievable. 
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Handout 11. Driver diagram for the Grove Elementary School 
example case 


NORTHEAST 
& ISLANDS 


Regional Educcional Laboratory 
ALEDC 


Driver diagram 


This completed driver diagram shows how the change practices (the interventions or specific 
work practices) are predicted to affect the secondary, and in turn, the primary driver. 


© Aim Statement 


Increase student engagement 
in math class 


The percentage of students 
performing at proficient or 
above will increase by 10%, 
and the gap between low- 
income students and their 
peers will decrease 


Improve teachers’ knowledge 
and use of best practices in 
math instruction 


© Primary Drivers 


© Secondary Drivers 


Increase student use of 
math discourse 


Increase the relevance of 
math to students’ lives 


Increase teachers’ capacity 


to build students’ conceptual 


understanding in math 


Provide math professional 
development for teachers 


Supplement the math 
curriculum with additional 
resources and tools 


© Change Practices 


Teachers use “sentence 
starters” to prompt math 
discourse 


Teachers implement 
conversation protocol to 
promote active listening 


Teachers develop and use 
examples that are relevant 
to students’ lives 


Math specialist provides 
professional development 
and support for 
implementing rich tasks 


Teachers employ 
observational rounds and 
provide feedback 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 
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Meeting 3 
Goals 


e Review the draft fishbone diagram developed in Meeting 2, make any needed changes, and confirm consensus 
on the specific problem of practice and key factors. 


e Establish the aim for the project and draft a driver diagram. 


Agenda (120 minutes) 


Materials 
Welcome and review of agenda and goals 5 minutes 
Review and revise the draft fishbone Draft fishbone diagram developed in Meeting 2 35 minutes 
diagram developed in Meeting 2 
Watch Video 3 and discuss Handout 17— Video 3 (available at https://youtu.be/NqaVgGHrcqU), 15 minutes 
Driver diagram for the Grove Elementary | Handout 11—Driver diagram for the Grove Elementary 
School example case School example case (p. ||-28 and https://ies.ed.gov/ncee/ 


edlabs/regions/northeast/pdf/handout11_Driver-diagram- 
example.pdf) 


Activity: Develop a driver diagram Handout 12—Template for a driver diagram (p. II-32 and 60 minutes 
https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/ 
handout12_Driver-diagram-template.pdf), flip chart, sticky 

notes 


Wrap up and next steps 5 minutes 


Welcome and review of agenda and goals (5 minutes) 


Begin the meeting with a welcome and a review of the agenda and goals. The meeting will center on confirming 
the fishbone diagram and developing a driver diagram. 


Review and revise the fishbone diagram from Meeting 2 (35 minutes) 
Review the draft fishbone diagram developed in Meeting 2 and address any changes that should be made using 
Handout 10—Guiding questions for developing a fishbone diagram (p. ||-23) to support the discussion and the 


following questions: 


e Should the problem statement be changed? Is it too general or too specific? Is it important and timely for us to 
address? 


¢ Do the “major bones” or key factors directly relate to the problem? Are they contributing to the problem? 
Have we left out any key factors? 


e Are the causes connecting to each factor the right causes? Is anything missing? 
Watch Video 3 and discuss Handout 11—Driver diagram for the Grove Elementary School example case (15 minutes) 
Video 3 (available at https://youtu.be/NqaVgGHrcqU) explains what a driver diagram is and uses the Grove Ele- 


mentary School example case to illustrate how to develop one. 
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After watching the video, review Handout 11—Driver diagram for the Grove Elementary School example case (p. |I-28 
and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout11_Driver-diagram-example.pdf) and discuss. If 
read from right to left, are the relationships between the change practices, the drivers, and the aim clear and logical? 


Activity: Develop a driver diagram (60 minutes) 


Nol Kom Ko aal=\clulatemr-lellin-laele 


As with the development of the fishbone diagram, use a flip chart and sticky notes so 
Improvement Team members can brainstorm drivers and change practices. They can 
write each idea on a separate sticky note and post them to a blank driver diagram. 
This will enable team members to move the ideas around in real time and to discuss 
what drivers are related to one another and what change practices might influence 
specific drivers. 


Follow the four steps below to develop a driver diagram using the Improvement Team’s fishbone diagram as a 
reference and a flip chart and sticky notes or Handout 12—Template for a driver diagram (p. \|-32 and https://ies. 
ed.gov/ncee/edlabs/regions/northeast/pdf/handout12_Driver-diagram-template.pdf) to brainstorm drivers and 
change practices. Refer to the pre-meeting reading for a description of the aim, primary and secondary drivers, and 
change practices, and to Handout 11—Driver diagram for the Grove Elementary School example case (p. \I-28 and 
https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout11_Driver-diagram-example.pdf) for examples. Also 
refer to Handout 13—Guiding questions for developing a driver diagram (p. ||-33) while developing the diagram. 


Step 1. Develop the aim statement. Starting with the Improvement Team’s fishbone diagram, develop an aim 
statement based on the problem statement that the team generated for the fishbone diagram. Remember that 
the aim is the specific and measurable goal, developed in response to the problem statement, that guides the 
improvement effort. It should describe what the team wants to achieve, by how much the team wants to increase 
or decrease something, and, if possible and appropriate, by when. In the Grove Elementary School example case 
the problem statement is that students in elementary grades, especially low-income students, are below profi- 
cient in math. The aim statement in the driver diagram example case reflects this. The aim is: “The percentage of 
students performing at proficient or above will increase by 10 percent, and the gap between low-income students 
and their peers will decrease.” Similarly, consider how to turn your problem statement into a specific and measur- 
able aim for the continuous improvement effort. 


Step 2. Develop the primary drivers. The primary drivers directly affect the aim and should focus on changes that 
are essential to making the improvement described in the aim statement. A driver diagram generally has three to 
five primary drivers. 


Step 3. Develop the secondary drivers. The secondary drivers should derive from the primary drivers and should 
be specific leverage points that are expected to have a direct impact on the primary drivers. 


Step 4. Develop change practices. The change practices should derive from the drivers and should be specific, 
actionable, and measurable. In general, seek change practices that have evidence in support of their use. Some 
resources for identifying evidence-based practices can be found on the next page in Resources for identifying 
evidence-based change practices. 


Once the Improvement Team has developed a driver diagram, it is ready to embark on preparations for the first 
Plan-Do-Study-Act (PDSA) cycle, which will be the focus of Meeting 4. 
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Resources for identifying evidence-based change practices 


The following resources will be useful in learning more about evidence-based practices and how to identify them. This is not an 


exhaustive list, but it may be useful, depending on the type of drivers identified. Review these resources before Meeting 4 and 


come prepared to share any ideas about any change practices that may be appropriate to implement and test. 


Identifying and implementing educational practices supported by rigorous evidence (https://www2.ed.gov/rschstat/research/ 
pubs/rigorousevid/rigorousevid.pdf). This guide by the U.S. Department of Education provides user-friendly tools to distin- 
guish practices supported by rigorous evidence and provides a list of additional websites that can help identify evidence-based 
practices. 

Guidelines for identifying evidence-based interventions for school improvement (http://www.fcrr.org/essa/). These self-study 
guides help schools systematically consider the evidence base and appropriateness of school improvement interventions for 
use in schools. They may also help in implementing a continuous improvement effort. 

Non-regulatory guidance: Using evidence to strengthen educational investments. (https://www2.ed.gov/policy/elsec/ 
leg/essa/guidanceuseseinvestment.pdf). This resource provides state education agencies, local education agencies, schools, 
educators, and partner organizations with information to assist in selecting and using evidence-based activities, strategies, and 
interventions. 

What Works Clearinghouse (WWC) (https://whatworks.ed.gov/). The WWC reviews existing research on different programs, 
products, practices, and policies in education and provides information so that practitioners can make evidence-based 
decisions. 
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Handout 12. Template for a driver diagram 


/REL 


nommest Template for a driver diagram 


Regianol Educational Laboratory 
ALEDC 


Complete your own driver diagram beginning with the aim statement and working through 
primary and secondary drivers and change practices. Add connecting lines as appropriate. Then, 
check your logic from right to left: "If [change practice], then [secondary -> primary -> aim]." 


© Aum Statement © Primary Drivers © Secondary Drivers © Change Practices 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 
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Handout 13. Guiding questions for developing a driver diagram 
Along with your fishbone diagram, and the description of the different elements of the driver diagram, also con- 
sider the guiding questions below while developing and reviewing your driver diagram. Remember that the driver 
diagram helps in understanding where you’re going with your work; identifies a logical set of smaller, more tangi- 
ble goals; and supports the selection of specific actions or changes. 

Aim statement 

e What is your goal? 

e What are you trying to accomplish? 

¢ Does the aim statement describe what you want to achieve and by when? 

e Is the aim attainable and realistic? 

e Is the aim specific and measurable? 

e Is the aim statement clearly articulated? 

Primary drivers 

e Are the primary drivers directly related to the aim? 

e How do the primary drivers achieve the aim? Do they achieve the aim independently or together? 

¢ Do the primary drivers identify the conditions that are necessary and sufficient to achieve the aim? 
Secondary drivers 

e Are the secondary drivers expected to have an impact on the primary drivers? Which ones? 

e Are the secondary drivers defining specific leverage points that are expected to influence the primary drivers? 


Change practices 


e Are the change practices directly related to the secondary drivers (or to the primary drivers if there are no 
secondary drivers)? 


e Are the change practices specific and measurable actions? 
e Are the change practices realistic and achievable? 


e Can the change practices be implemented frequently over a short period? 
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Wrap up and next steps (5 minutes) 


It is unlikely that the Improvement Team will reach consensus on the driver diagram in a single meeting. So, it 
may be necessary to schedule an additional meeting to complete the activity. But regardless of where the team 
arrives by the end of the meeting, the final step of the meeting will be to confirm that the work generated during 
this meeting will be compiled, and a draft driver diagram will be developed and shared before moving to the next 
stage in the process. Determine how this will be completed and who will do it. 


Cl leader 


If an additional meeting is needed to complete the driver diagram, schedule this meeting to occur before 
Meeting 4. 


Send the completed draft driver diagram to the full team for review. (The Cl leader can designate another team 
member to complete this task.) Remind team members to use Handout 13— Guiding questions for developing a 
driver diagram (p. |I|-33) when reviewing it. 


Ask Improvement Team members to review Resources for identifying evidence-based change practices (p. II-31) 
and identify any change practices that may be relevant to the continuous improvement effort and to come to 
Meeting 4 prepared to discuss potential change practices for the first PDSA cycle. Potential change practices 
may be based on evidence-based practices, Improvement Team members’ own ideas, or other resources. 


Ask Improvement Team members to complete the pre-meeting reading for Meeting 4 on measurement for 
improvement and come prepared to discuss potential measures for the first PDSA cycle. 


Decide how the Improvement Team will complete Handout 15—Template for the Plan-Do-Study-Act planning 
tool (p. Il-44 and https://ies.gov/ncee/edlabs/regions/northeast/pdf/handout15_PDSA-template.pdf) during 
Meeting 4 (on paper or projected on a screen for the group to view together). 


Improvement Team members 


Review the draft driver diagram and Handout 13—Guiding questions for developing a driver diagram (p. |I-33) 
and come to Meeting 4 prepared to make any changes and move forward with planning for the first “Do” cycle, 
in which a specific change practice will be tested. 


Review Resources for identifying evidence-based change practices (p. Il-31) and come to Meeting 4 with any 
change practices that may be relevant to the continuous improvement effort. Potential change practices may 
be based on evidence-based practices, Improvement Team members’ own ideas, or other resources. 


Complete the pre-meeting reading for Meeting 4 on measurement for improvement and come prepared to 
discuss potential measures for the first PDSA cycle. 


Meet to finish drafting the driver diagram, if necessary. 


Data manager 


None. 


Recorder 


Send the notes from Meeting 3 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


Prepare to take notes for Meeting 4. 
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Meeting 4. Preparing for Plan-Do-Study-Act cycles 


The purpose of this meeting is to select the driver and related change 
practice that will be the focus of the initial Plan-Do-Study-Act (PDSA) 
cycle and to prepare to launch the cycle. After several PDSA cycles, the 
Improvement Team may decide to select a new driver or change practice 
and launch a new series of PDSA cycles, in which case the team would 
return to this meeting, Meeting 4, to prepare for a new PDSA cycle. 


Preparations 
Cl leader 


e Ask Improvement Team members to review the draft driver diagram 
developed in Meeting 3 and Handout 13—Guiding questions for devel- 
oping a driver diagram (p. |I-33) and come prepared to discuss any 
changes. 


e Ask Improvement Team members to review Resources for identifying 
evidence-based change practices (p. II-31) and identify any change 
practices that may be relevant to the continuous improvement effort 
and to come prepared to discuss potential change practices for the 
first PDSA cycle. Potential change practices may be based on evi- 
dence-based practices, Improvement Team members’ own ideas, or 
other resources. 


e Ask Improvement Team members to complete the pre-meeting 
reading (see next page) on measurement for improvement and come 
prepared to discuss potential measures for the first PDSA cycle. 


e Decide how the Improvement Team will complete 
Handout 15—Template for the Plan-Do-Study-Act planning tool 
(p. I-44 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/ 


Materials needed 


Draft driver diagram developed in 
Meeting 3. 

Pre-meeting reading: Understand- 
ing measurement for improvement 
(p. I-37) 

Handout 13— Guiding questions 
for developing a driver diagram 

(p. I-33) 

Handout 14—Plan-Do-Study-Act 
planning tool for the Grove Elemen- 
tary School example case (p. |I-43 
and https://ies.ed.gov/ncee/ 
edlabs/regions/northeast/pdf/ 
handout14_PDSA-example.pdf) 
Handout 15—Template for the 
Plan-Do-Study-Act planning tool 
(p. Il-44 and https://ies.gov/ncee/ 


edlabs/regions/northeast/pdf/ 
handout15_PDSA-template.pdf) 
Handout 16—Discussion guide for 


data collection planning (p. I|-47) 
Handout 17—Run chart templates 
and examples (p. II-51 and https:// 
ies.ed.gov/ncee/edlabs/regions/ 
northeast/xls/handout17_Run- 
chart.xlsx) 

Handout 18—Protocol for data 
dialogue (p. II-65) 


handout15_PDSA-template.pdf) during the meeting (on paper or projected on a screen for the group to view 


together). 


Improvement Team members 


e Review the draft driver diagram developed in Meeting 3 and Handout 13—Guiding questions for developing a 
driver diagram (p. ||-33) before the meeting and come prepared to make any changes and move forward with 
planning for the first “Do” cycle, in which a specific change practice will be tested. 


e Review Resources for identifying evidence-based change practices (p. II-31) and come with any change prac- 
tices that may be relevant to the continuous improvement effort. Potential change practices may be based on 


evidence-based practices, Improvement Team members’ own ideas, or other resources. 


¢ Complete the pre-meeting reading (see next page) on measurement for improvement and come prepared to 
discuss potential measures for the first PDSA cycle. 
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Data manager 
e None. 
Recorder 


e Send the notes from Meeting 3 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for the meeting, including a record of decisions on Handout 15—Template for the Plan- 
Do-Study-Act planning tool (p. \l-44 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout15__ 
PDSA-template.pdf). 
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Pre-meeting reading: Understanding measurement for 
improvement 


Measurement for improvement is different from measurement for accountability, which may be more familiar to 
school-based practitioners (Bryk et al., 2015). Measurement for accountability generally: 


e Focuses on outcomes or results. 
e Is used to make high-stakes decisions. 
¢ Does not provide information about how outcomes were achieved. 


In contrast, measurement for improvement, while also focused on outcomes, investigates a small set of change 
practices that may be implemented, studied, and refined to test a working theory of change in a particular 
context. 


Measures for improvement should be practical—that is, they should be: 


e Embedded in practitioners’ regular work in the process of teaching and learning. Ideally, those doing the 
“improving” are involved in selecting the measures. 


e Administered frequently to identify opportunities for change and to assess whether the measure is yielding the 
targeted outcomes. 


e Made accessible, in language, tone, and content, to those using the measures as well as to those making deci- 
sions based on the results. 


Types of measures 


Central to continuous improvement is practical measurement that is tied directly to the working theory of 
change. There are several types of measures. The first is the measurement of the aim, which is the ultimate 
targeted outcome of the continuous improvement effort. In the Grove Elementary School example case the aim 
is increased math achievement for all students. The aim is typically a long-term outcome. It is also important to 
develop measures related to the primary and secondary drivers that serve as intermediate outcomes and are 
aligned tightly with the tools, processes, or strategies enacted during the “Do” part of the Plan-Do-Study-Act 
(PDSA) cycle. 


There are three types of data to collect in a PDSA cycle (Bryk et al., 2015; Langley et al., 2009): 


¢ Outcome measures. Outcome measures are data on the intermediate outcomes related to the primary and 
secondary drivers and change practices. These measures are collected to track improvement that has resulted 
from implementation of the change practices from the baseline. A baseline is a measurement of relevant out- 
comes before the change practice is implemented. In the Grove Elementary School example case an outcome 
measure, tied to a driver, is improvement in students’ participation in class discussion. Improvement in this 
outcome is determined based on observing an increase in student participation compared with the baseline 
measure of student participation taken before the change practice was implemented. Effective outcome mea- 
sures provide information quickly enough to allow teams to make appropriate adjustments if progress is not 
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observed. In the Grove Elementary School example, students’ grades at the end of the term might be aligned 
to the larger problem teachers are trying to address, but they are too far removed from the specific change 
being implemented. The percentage of students each day who use sentence starters that their teachers have 
modeled would be a better outcome measure. It is both timely and aligned to the change practice. 


e Process measures. Process measures are data on whether the steps associated with implementing the change 
practice took place as intended. For example, did teachers implement the sentence starter change practice 
as intended? If not, why not? In the Grove Elementary School example case a process measure could be the 
number of times teachers used sentence starters in their lessons each day, as measured by the teachers 
keeping classroom logs of their use of the sentence starters. 


e Balance measures. Balance measures are data on the unintended consequences and the costs of the change 
practice. This information is collected to ensure that improvement in one practice does not have a nega- 
tive impact on other practices or outcomes and to determine whether the costs are in proportion to, or “in 
balance” with, the benefits of the change practice. In the Grove Elementary School example case a balance 
measure could be the records of how much time teachers devoted to introducing the sentence starters to 
students during math instruction time. 


Measures are deemed practical when practitioners can collect, analyze, and use them in their daily work. The 
focus is on collecting just enough of the right data that will inform practitioners whether an improvement has 
occurred without overburdening them in the collection process. Determining how to collect data that are both 
useful and reasonable for practitioners to collect in the classroom can require several cycles to iron out. In the 
Grove Elementary School example case the teachers had to modify their data collection strategies after the initial 
cycle to make it feasible for them to collect the necessary data. While they initially thought each teacher could 
collect his or her own data on student participation, it proved difficult to do so while teaching. Instead, the teach- 
ers involved others to observe and collect the data. The kindergarten teachers combined classrooms a few times 
so they could observe one another and collect data, and the math coach served as an additional data collector, 
observing several classes. It is okay if it takes a few tries to arrive at a good approach to practical data collection; 
that is part of the process and one of the reasons that multiple cycles are implemented. 


Run charts 


Run charts are a tool often used to chart data over time. They can be used to examine trends and patterns in both 
outcome and process data. In a typical continuous improvement run chart the horizontal axis represents time 
and the vertical axis represents the measure of interest, such as the percentage of students who used sentence 
starters. Enough time points, perhaps 8-10, are needed to allow for normal variation. With simple run charts, the 
implementers of change practices can record data in real time, simplifying data entry and reporting. 


The two run chart examples on the next page display outcome and process data from the Grove Elementary 
School example case. Handout 17—Run chart templates and examples (p. |I-51 and https://ies.ed.gov/ncee/ 
edlabs/regions/northeast/xls/handout17_Run-chart.xlsx) includes several Excel templates for creating run charts, 
with instructions on how to use them. Use these templates or create your own run charts to track the Improve- 
ment Team’s data over time. 
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Run chart example 1: Outcome measure for teachers’ use of sentence starters 


Percent of students that used sentence starters 


Nov.26 Nov.27 Nov.28 Nov.29 Nov.30  Dec.4 Dec. 6 Dec.7 Dec.10 Dec. 11 


Run chart example 2: Process measure of implementation of change practice 


Number of sentence starters used by teacher 
6 


Nov.26 Nov.27 Nov.28 Nov.29 Nov.30  Dec.4 Dec. 6 Dec.7 Dec.10 Dec. 11 


p 


Ww 


N 


= 
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Meeting 4 

Goals 

e Select a driver and a change practice for the first PDSA cycle. 

e Select measures for the first PDSA cycle and create a data collection plan. 


e Discuss next steps in launching the first PDSA cycle. 


Agenda (120 minutes) 


Materials 
Welcome, review of agenda and goals, and Draft driver diagram, Handout 13—Guiding questions | 10 minutes 
discussion of driver diagram and pre-meeting | for developing a driver diagram (p. lII-33) 
reading 
Activity: Select a driver and change practice for | Draft driver diagram developed in Meeting 3 35 minutes 


the first Plan-Do-Study-Act cycle 


Introduction to the Plan-Do-Study-Act planning | Toolkit, Handout 15—Template for the Plan-Do-Study- | 15 minutes 

tool and data discussion Act planning tool (p. I-44 and https://ies.ed.gov/ 
ncee/edlabs/regions/northeast/pdf/handout15_PDSA- 
template.pdf), Handout 18—Protocol for data dialogue 
(p. I-65) 


Watch and discuss Video 4 Video 4 (https://youtu.be/KN3702rzuMc) 10 minutes 


Activity: Select measures and develop a data Toolkit, Handout 15—Template for the Plan-Do-Study- | 40 minutes 

collection plan for the first Plan-Do-Study-Act | Act planning tool (p. \|-44 and https://ies.ed.gov/ 

cycle ncee/edlabs/regions/northeast/pdf/handout15_PDSA- 
template.pdf) 


Wrap up and next steps 10 minutes 


Welcome, review of agenda and goals, and discussion of driver diagram and pre-meeting reading (10 minutes) 
Begin the meeting with a welcome and a review of the agenda and goals. Have a brief discussion to: 

e Address any changes to the draft driver diagram. 

e Review the pre-meeting reading on measurement for improvement. 

For the driver diagram review, use Handout 13—Guiding questions for a driver diagram (p. \|-33) to confirm the 
driver diagram addresses all the ideas. Also try following the logic of the driver diagram from right to left: “If we 
do x change practice, we will achieve y driver; and if we achieve y and z drivers, we will reach our aim.” Look for 
the logic within the driver diagram: Do the change practices and drivers “add up” to the aim? 


For the review of the pre-meeting reading, focus the discussion on: 


e The features of measurement for improvement and how they differ from those of measurement for other 
purposes, such as accountability. 


e The three types of measures for PDSA cycles: outcome, process, and balance. 
e The characteristics of good measures for improvement. 
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Activity: Select a driver and change practice for the first Plan-Do-Study-Act cycle (35 minutes) 


Use the Improvement Team’s completed driver diagram to prioritize which driver will be the focus of the first 
PDSA cycle and what change practice the team will test first. Over time the team will probably test several change 
practices across multiple drivers; for the first cycle select a single driver and change practice. The driver should be 
one that is both high leverage and accessible. That is, implementing the change practice must have the potential 
for advancing progress toward the aim, and the driver must be within the purview of the team members who will 
implement the change practice. 


In the Grove Elementary School example case the teachers identified the primary driver “increase student engage- 
ment” and the related secondary driver “increase student use of math discourse.” They focused on these related 
drivers because addressing these drivers was something they felt had the potential to yield positive results for 
students—that is, it is high leverage—and because making changes to classroom practice was something they 
could do—that is, it is accessible. The specific change practice, related to the secondary driver, was to implement 
the use of sentence starters in math class; this was a narrow change practice that could be implemented quickly 
and without substantial resources. 


In selecting a driver and a change practice, the Improvement Team should discuss the following questions and be 
able to answer them affirmatively: 


e Is the driver likely to be high leverage, in that achieving the driver might yield positive results related to the 
overall aim? Is the driver something accessible, in that it is within the control of the Improvement Team 
members? 


e What is the specific change practice? How and how often will it be implemented? 


e Is the “grain size” of the related change practice appropriate? Is it narrow enough to allow for immediate 
implementation? 


e Is implementing the change practice feasible? Are there enough resources, and do Improvement Team 
members have the knowledge and skills to support implementation? 


e If the Improvement Team members who will be implementing the change practice do not have all the knowl- 
edge and skills to implement it right now, can they acquire the knowledge and skills quickly with training and 
resources? Are these training and resources available? 


e Are there enough people available to implement the change practice? There is no set number of people that 
must implement the change; in some cases 3—4 teachers would be appropriate, whereas in others 10-15 would 
be. It is more important to have a few people who are truly ready and willing to implement the practice and 
collect the data than it is to have a large group of people engaged in implementing the change practice, espe- 
cially if they are not ready or willing to do so. 


e Is data collection associated with the change practice feasible? Can you quickly and easily collect outcome and 
process measure data? 


¢ Do you have appropriate tools or protocols to support data collection, such as simple observation instruments, 
quick surveys, or other tools that support teachers or others to easily collect the relevant data? 


e Is the change practice aligned with the aim the Improvement Team identified? 


After discussing these questions and identifying the driver and change practice, have the recorder or meeting 
facilitator record the decision on Handout 15—Template for the Plan-Do-Study-Act planning tool (p. \l-44 and 
https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout15_PDSA-template.pdf), which is introduced next. 
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Introduction to the Plan-Do-Study-Act planning tool and data discussion (15 minutes) 


Take a few minutes to review Handout 14—Plan-Do-Study-Act planning tool for the Grove Elementary School 
example case (p. Il-43 and _ https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout14_PDSA-example. 
pdf) and Handout 15—Template for the Plan-Do-Study-Act planning tool (p. \|-44 and https://ies.ed.gov/ncee/ 
edlabs/regions/northeast/pdf/handout15_ PDSA-template.pdf). The tool will help the Improvement Team docu- 
ment the change practices and the data collection activities for the “Study” part of the PDSA cycle. 


To complete the template (see example on the next page): 


e Record the “Plan” title—a succinct name for the first PDSA cycle—related to the driver and the change practice 
selected. 


e Determine the cycle timeframe and record it in the appropriate fields on the template. 

e Record the selected driver and change practice, including who will implement, in what contexts, and when. 
Note the key decision points in the template. After deciding on the relevant driver and related change practice 
and the start and end date of the cycle, determine who will implement the change practice. Then determine the 


appropriate data to collect to understand whether the change practice is being implemented as planned and is an 
improvement related to the driver and aim. 
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Handout 14. Plan-Do-Study-Act planning tool for the Grove 
Elementary School example case 


SREL 


NORTHEAST 
& ISLANDS 


fey 


Plan Title: Teachers engage students in mathematical discourse 
Increase student engagement in mathematics/increase student use of mathematical discourse 


Relevant Driver: 


Cycle 1 start date: 


11/26/19 


ad DDsyAWo)F-Jalaliay-a Kole) mice) mm aat-M-y €10n) 0) (-Mot-1i-) 


Cycle end datel 2/21/19 


Overall Aim: Improve mathematics achievement for all students and reduce gap between low-income and non low-income students 


Change Practice 


Teachers implement the use of sentence 


Who implements 


starters at least two times a week in math class. 


Question 


Process 

How well was 
the change 
practice 
implemented? 


This document 


e 
:REL 
NORTHEAST 
& ISLANDS 


Question 
Outcome 
(Driver) 

Did the change 
practice result 
inan 
improvement 
related to the 
associated 
driver? 


Balance 

Were there any 
unintended 
negative 
consequences 
that resulted 
from 
implementing 
the change 
practice? What 
are the costs 
associated with 
this change 
practice? 


a part of the Continuous Improvement in Edu 


Data Collector(s) 
Who will collect 
data? 


Participating 
teachers (7) 


Instrument(s)/Measures(s) 
What data will be collected? 


Teacher log to record the use 
of sentence starters (date of 
instruction, minutes spent 
planning and implementing) 
and reflections on how it went 
(barriers encountered, 
adaptations to planned 
implementation) 


Jen Nguyen 
(Cl team 
member) will 
observe each 
participating 
teacher once 


Observation protocol to assess 
how well teachers implemented 
sentence starters (instructional 
minutes spent on discussion 
using sentence starters, explicit 
instructions on using sentence 
starters, checklist of types of 
starters used) 


Data Collector(s) 
Who will collect 
data? 


Participating 
teachers (7) 


Instrument(s)/Measures(s) 
What data will be collected? 


Student participation log (to 
track which students used 
sentence starters and how 
often) 


Participating 
teachers (7) 


Teacher logs including date 
of instruction, minutes spent 
planning and implementing, 
and reflections on how it 
went (e.g., barriers 
encountered, adaptations to 
planned implementation), 
and if they had to sacrifice 
other activities for the sake of 
implementing the change 
practice. Same log as above. 


Selected teachers 


ed by the Regional Ei 


Implementing in what context 


4th and 5th grade 


mathematics instruction 


Timing 
Sample When will 
Fromwhomwill data be 


data be collected? collected? 
All Log 
participating completed 
teachers (7) on each 


day of 
sentence 
starter use 
One math Week 2 or 
lesson of 3 of cycle 
each 
participating 
teacher 


Sample Timing 

From whom will = When will 

data be data be 

collected? collected? 
Baseline 

All students moon’ 

in collected before 

i ninati th f 

Participating sentence starters 

teachers’ to track which 
students 

classes participate in 
math discussion 
and how often. 
Subsequent 
measures at the 
time of instruction 
using sentence 
starters. 

All Log 

participating completed 

teachers (7) on each 
day of 
sentence 
starter use 


Data Entry/ 
Reporting 
How will data 
get compiled? 


Entry into 
Excel 
spreadshee 
t, by 
teacher by 
the end of 
day 


Real time 
entry into 
Excel 
spreadshee 
t by 
observer 


Data Entry/ 
Reporting 
How will data 
get compiled? 


Log completed 
‘on hard copy or 
Excel sheet. 
Summary 
entered by the 
end of day in 
Excel: date, list 
of present 
students, and 
notation of who 
used a sentence 
starters and how 
many times 


Entry into 
Excel 
spreadshee 
t, by 
teacher by 
the end of 
day 


When implementing 
November-December 


Predictions 

What do we think 
will happen (what 
will the data tell us)? 


At least 6 (of 7) 
teachers will use 
sentence starters 
at least twice a 
week across the 
4-week cycle. 


Results 
(Completed after 
data collection) 


At least 6 (of 7) 
teachers will 
provide explicit 
instruction on at 
least 3 types of 
sentence starters 
for a class 
discussion lasting 
at least 10 minutes 


cational Laboratory Northeast & Island: 


ed DsyAWo)F-Talaliay-a Kole) me) mm aa(-M => €10n) 0) (-Mor-1x-) 


Predictions 

What do we think 
will happen (what 
will the data tell us)? 


Results 
(Completed after 
data collection) 


The number of 
students 
participating in 
math 
discussions will 
increase from 
baseline to the 
end of the cycle. 


On average, 
teachers will 
spend less than 
10 minutes 
planning for 
instruction for 
the use of 
sentence 
starters and at 
least 10 minutes 
doing the activity 
in class. 


This document was prepared under Contract ED-IES-17-C-0008 by Regional Educational Laboratory Northeast & Islands, administered by Education Development 
Center. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial 
products, or organizations imply endorsement by the U.S. Government. 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 
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Handout 15. Template for the Plan-Do-Study-Act planning tool 


A fillable PDF of the template is available at https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout15 __ 
PDSA-template.pdf. 


REL PDSA Planning Template 


& ISLANDS 


Aesjoa acon erate 
wat 


Plan Title: Cycle 1 start date: Cycle end date: 


Relevant Driver: 


Overall Aim: 
Change Practice Who implements Implementing in what context When implementing 
Timing Data Entry/ Predictions 
Data Collector(s) Sample When will Reporting What do we think Results 
Question Instrument(s)/Measures(s) Whowill collect Fromwhomwill data be Howwilldata will happen (what (Completed after 
What data will be collected? data? data be collected? collected? get compiled? _ will the data tell us)?__data collection) 
Process 
How well was 
the change 
practice 
implemented? 


part of the Continuous Improvement in Education Tool Educational Laboratory Northeast & Islands. 


SREL 


NORTHEAST 
& ISLANDS 


Reso ctl aaa 
1 


Sample Timing Data Entry/ Predictions 
Data Collector(s) From whomwill When will Reporting What do we think Results 
Instrument(s)/Measures(s) Who will collect data be data be Howwilldata will happen (what (Completed after 
Question What data will be collected? data? collected? collected? get compiled? will the data tell us)?_—_data collection) 
Outcome 
(Driver) 
Did the change 
practice result 
inan 
improvement 
related to the 
associated 
driver? 


Balance 

Were there any 
unintended 
negative 
consequences 
that resulted 
from 
implementing 
the change 
practice? What 
are the costs 
associated with 
this change 
practice? 


This document was prepared under Contract ED-IES-17-C-0008 by Regional Educational Laboratory Northeast & Islands, administered by Education Development 
Center. The content does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial 
products, or organizations imply endorsement by the U.S. Government. 


is a part of the Continuous Improvement in Education Tool Educational Laboratory North 
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Watch and discuss Video 4 (10 minutes) 


Video 4 (available at https://youtu.be/KN3702rzuMc) illustrates how to use a driver diagram to generate appro- 
priate data for continuous improvement, using the Grove Elementary School example case. After viewing the 
video, discuss how outcome, process, and balance measures differ from one another. 


Activity: Select measures and develop a data collection plan for the first Plan-Do-Study-Act cycle (40 minutes) 


Take a moment to review the three types of measures (outcome, process, and balance) in Handout 14— 
Plan-Do-Study-Act planning tool for the Grove Elementary School example case (p. ||-43 and https://ies.ed.gov/ 
ncee/edlabs/regions/northeast/pdf/handout14_PDSA-example.pdf). Consider how the examples relate to your 
own driver diagram and note the following about data collection: 


e The data collected for the PDSA cycle in the Grove Elementary School example case include a participation 
log and observation data. Collecting these more specific and frequent data can help the Improvement Team 
understand whether the change is being implemented as intended (process measures), whether it is yielding 
the immediate results defined in the related driver (outcome measures), and whether there are other, unex- 
pected outcomes associated with implementation (balance measures). This information will inform decisions 
about the next test of the change practice. For example, if the student participation log shows that when 
teachers use sentence starters, math discourse improves for only some students and not for students about 
whom the teachers were most concerned, the team might consider changing the implementation of sentence 
starters or implementing a different change practice to improve the math discourse of low-income students in 
particular. 


e The data collected for the PDSA cycle in the Grove Elementary School example case do not include data related 
to the overall aim, which is annual math achievement. This is because standardized math achievement results 
are collected too infrequently to inform a PDSA cycle. In fact, the Improvement Team may hypothesize that 
improving math discourse in the classroom will improve student achievement; however, the team will not 
know this after a single or even several cycles. What it will know after several cycles is whether implementing 
the change practice improves student discourse, which is an intermediate outcome related to the driver. The 
team, as graphically represented in the driver diagram, hypothesizes that increased student discourse improves 
student achievement. This relationship may be grounded in empirical evidence conducted in another setting 
or based on practitioner experience or supposition. When the next annual student math achievement data 
are available, there may be an improvement in student achievement. This improvement in the overall aim may 
thus increase confidence among team members and other educators that the increase in student discourse is 
related to positive achievement outcomes. 


¢ Outcome data are needed that relate more directly to the driver, such as a measure of student use of sentence 
starters, and that are collected more frequently—in this case, daily or weekly. The Improvement Team may 
hypothesize that improving math discourse in the classroom will improve student achievement; however, the 
team will not know this after a single or even several cycles. What it will know after several cycles is whether 
implementing the change practice improves student discourse, which is an intermediate outcome related to 
the driver. The team, as graphically represented in the driver diagram, hypothesizes that increased student 
discourse improves student achievement. This relationship may be grounded in empirical evidence conducted 
in another setting or based on practitioner experience or supposition. When the next annual student math 
achievement data are available, there may be an improvement in student achievement. This improvement in 
the overall aim may thus increase confidence among team members and other educators that the increase in 
student discourse is related to positive achievement outcomes. 
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e The use of predictions in the PDSA planning tool are important because the Improvement Team lists what it 
anticipates the outcomes will be for each cycle of implementation as a way to chart the expected versus actual 
improvement (if any) that is witnessed. As described above, realizing the predicted outcome builds confidence 
in the change practice and suggests directions for future implementation. Similarly, not realizing a predicted 
outcome might warrant the team reconsidering whether the change practice is likely to achieve the targeted 
outcomes. 


Directions for completing the template for the Plan-Do-Study-Act planning tool 


Step 1. Discuss data collection for outcome, process, and balance measures. Use Handout 16—Discussion guide 
for data collection planning (p. \|-47) to discuss the relevant data sources for the outcomes related to the driver 
and the change practice, as well as the processes and activities required to collect high-quality data. It includes 
example data sources to generate ideas. Here are some recommendations regarding the selection of measures 
and data collection strategies: 


e Ensure that the measures address the specific change practice that will be introduced and tested in this first 
PDSA cycle. Planning the measurement strategies at the same time that the Improvement Team selects the 
change practice will allow the team to confirm that the change practice is readily measurable. 


e Plan data collection to align with the daily activities of practitioners. If teachers are expected to collect data, 
consider the burden placed on them and ensure that the data collection strategies are manageable and realis- 
tic. Practical measurement should be as fast and simple as possible, ideally without requiring practitioners to 
greatly change their normal routines. 


e Collect data that are relevant and useful to the practitioners. The people collecting and analyzing the data 
should clearly understand why the data are being collected and how the data will inform practice. 


e Leverage relevant data already being collected in classrooms, meetings, or schoolwide, and consider whether 
any of them could be meaningful measures for this PDSA cycle. 


Step 2. Record the measures selected. After discussing the data collection, complete the section of Handout 15— 
Template for the Plan-Do-Study-Act planning tool (p. \|-44 and https://ies.ed.gov/ncee/edlabs/regions/northeast/ 
pdf/handout15_PDSA-template.pdf) that outlines the outcome, process, and balance measures with the decisions 
the Improvement Team made. Detail the data collection plan, including who will collect the data, how often, and 
using what format. A clear, detailed plan will promote consistency in how data are collected. 


Step 3. Make predictions. Record both predictions and the results of the data collection in Handout 15—Template 
for the Plan-Do-Study-Act planning tool (p. |l-44 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/ 
handout15_PDSA-template.pdf). Making predictions is an essential component of testing a change practice. 
Predictions are tied to the working theory of change. If the results do not confirm predictions, the Improve- 
ment Team will need to determine whether the theory of change is incorrect, the change practice was not 
well implemented, or the measure did not capture a true improvement. The team’s assessment of the reasons 
the predictions were not accurate may determine whether the change practice is adapted or discarded. See 
Handout 14—Plan-Do-Study-Act planning tool for the Grove Elementary School example case (p. \|-43 and https:// 
ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout14_PDSA-example.pdf) for reference. 
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Handout 16. Discussion guide for data collection planning 


Use these questions to discuss options for data collection, and record decisions on Handout 15—Template for the 
Plan-Do-Study-Act planning tool (p. \|-44 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout15_ 
PDSA-template.pdf). 


e What types of data are needed for outcome, process, and balance measures (see Examples for data collection 
planning below for ideas)? 


e What tools, protocols, or instruments are most relevant and practical for collecting the data needed? Exam- 
ples include assessments, surveys, checklists, observation forms, logs or journals, interviews, and attendance 
sheets. 


e What existing data sources or instruments should be considered for measurement in this cycle? Examples 
include lesson plans, walkthrough protocols, and formative assessments. 


e What new data sources or instruments should be considered? Examples include “do now’s” or student exit 
tickets and interview or survey questions. 


e When and how frequently will data need to be collected? 


e Who will be responsible for collecting the data? A teacher, principal, district staff member, or student? What 
training and instructions related to data collection will be needed? 


e How will data be compiled and prepared for the data team to review? Will hard-copy data need to be entered 
electronically? If so, how will this happen? What software or templates will be used? Who is responsible for 
ensuring that everyone’s data are submitted and prepared for presentation to the Improvement Team? 


Examples for data collection planning 


Type of measure Example data sources and methods 


Outcome (related tothe ¢ Progress monitoring assessments 


aim, drivers, or change e Unit or chapter tests 
practice) e Standardized achievement tests (most appropriate related to the aim rather than the 
drivers) 
e Counts to record such information as how many and which students participated in 
discussion 


e Ratings to measure the quality or frequency of phenomena such as student reports of 

confidence in participating in class discussions 

“Do now’s” or entry or exit tickets to gather responses to a small number of questions 

Surveys of teachers, administrators, students, or family members 

Interviews or focus groups of family members, students, teachers, or administrators 

Attendance or discipline records 

Behavioral checklists completed by teacher, parent, or student to document instances of 

appropriate or inappropriate behaviors 

Nonacademic assessments to assess areas such as social-emotional skills 

e Student work samples 

e Observations conducted by peer teachers, administrators, or others to assess such areas 
as student engagement, social interactions, and time on task 

e Written logs or gradebooks to record instances of behaviors such as whether students 
turned in homework 
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Type of measure Example data sources and methods 


Process (to determine e 
whether change practice 
was implemented 

with fidelity and to 
understand context of 
implementation) 


Balance (to identify e 
costs and unintended ° 
consequences of the 

change practice) e 


Lesson plan surveys to assess instructional activities such as the frequency of practice 
implementation 

Journals, such as for self-reflection on what worked or what did not work 

Classroom observations 

Administrator walkthroughs 

Checklist of actions completed 

Interviews or focus groups to collect detailed information on how implementation went 
Logs to record instances of a change practice implemented 

Administrative records such as class lists to determine characteristics of students who 
received the change practice 


Logs or journals for implementers to record time spent, challenges, and unexpected results 
Interviews or focus groups of administrators, teachers, or others involved in the change 
practice 

Surveys to collect specific information on costs, challenges, and fiscal data 
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Wrap up and next steps (10 minutes) 


Conclude the meeting with a review of what is going to occur during the first PDSA cycle: 


Review the change practice and what high-quality implementation of the change practice entails to ensure a 
common understanding. 


Record who is going to be testing the change practice and how often the change practice is going to be tested. 
For example, all math teachers will test the math discourse strategy at least twice per week. 


Review the data collection plan and assign responsibility for creating any of the measurement tools. For 
example, the math instructional coach is going to create the discourse participation log by the end of this week 


for teachers to use starting next week. 


Set the next PDSA meeting date and time to correspond with the end of the cycle. 


Cl leader 


Follow up with the Improvement Team members responsible for implementing the change practice and col- 
lecting data. 


Disseminate the materials needed to implement the change practice. 
Communicate with implementers about the timing of change practice implementation and data collection. 
Check in regularly with implementers and data collectors to ensure that the change practice and data collec- 


tion are occurring as planned and to determine whether additional resources and supports are needed. 


Work with the data manager to develop a plan for monitoring data quality and ensure that all data are collect- 
ed, summarized, and presented in an accessible way for discussion. 


Decide and confirm who will facilitate the data discussion in Meeting 5. 


Ask Improvement Team members to complete the pre-meeting reading for Meeting 5 on examining data and 
making decisions based on results. 


Improvement Team members 


Implement the change practice. 
Communicate with the meeting facilitator and data manager about progress, challenges, and support needs. 
Collect and submit data. 


Complete the pre-meeting reading for Meeting 5 on examining data and making decisions based on results. 


Data manager 


¢ Develop or identify tools for compiling data. 
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e Consider using the run chart instructions and templates provided with this toolkit (Handout 17—Run chart tem- 
plates and examples; see next page). 


¢ Develop instructions and procedures for compiling data. This could be a spreadsheet or shared form for data 
collectors to enter data and the procedures and timeline for document data submission. 


e Develop a process for managing data, such as a checklist of data expectations or procedures for merging data 
submitted by individuals. 


e Ensure that all data are collected, summarized, and presented in an accessible way for discussion during 
Meeting 5. 


Recorder 


e Send the notes from Meeting 4 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for Meeting 5. 
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Handout 17. Run chart templates and examples 


The Excel workbook associated with this toolkit (available at https://ies.ed.gov/ncee/edlabs/regions/northeast/ 
xls/handout17_Run-chart.xlsx) contains six worksheets that offer a series of templates and examples for different 
run charts. Each worksheet is in its own tab, shown at the bottom of the screen. The screenshot below points out 
the first two tabs: run chart template 1, which is the first template for a single process measure, and run chart 
example 1, which is the corresponding example. 


ek RUN CHART TEMPLATE: single process measure 
2 the green boxes 
| 3 
4 |Time point 
5 |Process measure 
| 6 
7 
| 8 Process measure 
9 1.2 
| 10 
11 ‘ 
| 12 
0.8 
13 
14 06 
15 
16 0.4 
Ly. 
18 0.2 
19 5 
20 
21 
22 Second tab is an example 


23 First tab is a template using the template 


| 24 
25 
| 26 


run chart template 1 run chart example 1 run chart template 2 run chart example 2 run chart example 3 participation log + 


I-51 


All the templates in the workbook have some cells shaded in green. Input your data in the green cells. Row 4 
(“Time point”) is where you document the date of the observations. Row 5 (“Process measure”) is where you 
input the data collected as your process measure. 


A B ic D E F G H il K L M 
RUN CHART TEMPLATE: single process measure 
the green boxes 


Time point 
Process measure 


Process measure 
Fill in your data here in 
the green boxes 


ee ae ae oe ee 
00) | Sif Oy [On | fs) 0] NS [ea op SO OP |S) ue | eo re 


| 19 


Your run chart will appear here 


IN NNNNNNN 
IN DU BP WNP OO 


run chart template 1 run chart example 1 run chart template 2 run chart example 2 run chart example 3 participation log - 


When you type your data into the green boxes, a run chart reflecting the data you entered will appear in the 


blank graph area. You can look at the second tab, run chart example 1, to see what this will look like with some 
sample data. 


The templates currently contain enough room for 10 time points. There is no rule for how many data points 
should be included in the run chart, but 10 data points is generally a good aim, depending on the data being 
collected and the length of the cycle. If you have more data than that, the run chart will not automatically update 
with the additional data points. You can edit the chart and select a larger data range if that is needed for your 
data set. (If you are unsure about how to do this in your version of Excel, type “select data” in the Excel help box.) 
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As you can see in the example below, whatever you type in for the time point data will show up in the run chart. 


The time points can be descriptive as in this example (“Day 1,” “Day 2”) or they can be actual dates or times. 


A B c D E E G H J K 


Be 
REE we©anauawne 


RR 
wr 


Time point 
Process measure 


RUN CHART EXAMPLE: One process measure over ten days 
he green boxes 


Process measure 


BR 
uf 


Time point data in chart 
corresponds to Inputs of row 4 


WNNNNNNNNNNP EB E 
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i) 
3 


4 > 


Day1 Day2 Day3 Day 4 Day 5 Day6 Day7 Day8 Day9 Day 10 


run chart template 1 run chart example 1 run chart template 2 run chart example 2 run chart example 3 Participation log 
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The worksheets run chart template 2 and run chart example 2 have charts for both an outcome and a process 
measure over time. There are two charts and an additional row of data, as you can see in the screenshot below. 


A B c D E F G H | J K L M N 


RUN CHART TEMPLATE: Outcome and process measures over time 
the green boxes 


Time point 
Outcome measure Two measures 
Process measure 


Outcome measure 


WAN DU FWN PR 


a 
ro 


100% 


RR 
wn 


80% 


PRR 
au S 


60% 
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4 > run chart template 1 run chart example 1 run chart template 2 run chart example 2 run chart example 3 participation log 
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The worksheet run chart example 2 includes an outcome measure that is a percentage and a process measure 
that is a whole number, as shown below. Run chart template 2 also has these defaults for the type of data expect- 
ed (for example, a percentage or a whole number), but you can change it if that is not appropriate for your spe- 
cific data. (Highlight the cells of data that are the wrong format and go to Format — Cells in the Excel menu.) The 
charts are lined up one on top of the other to help you visually identify any relationships or associations between 
the outcome and the process measures over time. 


A B Cc D E F G H I J K L M N 

al RUN CHART EXAMPLE: Outcome and process measures over time 

2 the green boxes 

3 

4 |Time point 

5 |Outcome measure 

6 |Process measure 

7 

8 Outcome measure 
9 60% 

10 

2 ” 

13 

t4 40% 

15 

16 30% 

T7: 

18 20% 

19 

20 10% 

21 

22 0% 

23 Day1 Day2 Day3 Day4 DayS Day6 Day7 Day8 Day9 Day 10 
24 

25 Process measure 
26 7 

27 2 

28 

29 : 

30 4 

31 ¥ 

32 

33 : 

34 1 
35 

Oo 
36 Day1 Day2 Day3 Day4 DayS Day6 Day7 Day8 Day9 Day 10 
37 
38 
39 
4 > run chart template 1 run chart example 1 run chart template 2 run chart example 2 run chart example 3 participation log + 
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The worksheet run chart example 3 also has both an outcome measure (percent of students that used sentence 
starters) and a process measure (number of sentence starters used by teacher). In this example the names of 
these measures are shown in the run charts. You can also see that the time points are actual dates. 


A B c D E FE G H I J K 
; lm aaa RUNCHARTEXAMPLE:Studentuseofsentencestarters. i (asi‘CSt;*”*”*”*”™”!”!OOO!OOOOO 
(Upaatedatalsing the login the "participation log" tab 
| 
\Time point Nov. 26 Nov. 27 Nov. 28 Nov. 30 Dec. 3 Dec. 4 Dec. 6 Dec. 7 Dec. 10 Dec. 11 
\Percent of students 
Ithat used sentence 
|starters 21% 36% 50% 31% 38% 53% 36% 33% 40% 42% 
|Number of 
\sentence starters 
lused by teacher 5 4 3 4 2 3 2 2 2 1 


Percent of students that used sentence starters 
60% 


50% 
40% 
30% 


20% 


Now. 26 Nov. 27 Nov. 28 Nov. 30 Dec. 3 Dec 4 Dec 6 Dec 7 Dec 10 Dec 11 


Number of sentence starters used by teacher 


3 

2 

UJ 
Nov. 30 Dec 3 Dec 4 Dec 6 Dec 7 


Nov. 26 Nov, 27 Nov. 28 Dec 10 Dec. 11 
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One main difference in run chart example 3 is that all the data being displayed in the run charts were not typed 
in on this worksheet—rather, these data are being pulled automatically from the last tab, participation log. The 
participation log (see below) provides two weeks’ worth of space to enable the user to complete a log about 
student and teacher use of sentence starters. Information that is put into the log within the outlined boxes will be 
tallied automatically, and the sheet will calculate the percentage of students in the class (subtracting absences, 
which are also notated) that used the sentence starters. These numbers (the calculated percentage of students 
using the sentence starters and the number of sentence starters the teacher used) are automatically updated in 
the other tab, run chart example 3, where the chart is also automatically updated. This allows the user to input 
the numbers only once in the log and then see the charts without extra effort. 


A B c D E E 


H I x K 


PARTICIPATION LOG EXAMPLE: For each data collection day, mark if student used sentence starter ("X") or was absent ("A") 


data within the boxes 


Nov. 26 Nov. 27 Nov. 28 Nov. 29 Nov. 30 
Student A x x x 
Student B x 
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You can fill in names and Xx X X 
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Total Week 
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"X" for when students used the 
sentence starter; "A" for absent, and 
number of sentence starters used by 
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Meetings 5-8. Examining results and related data 


This is the first meeting where Improvement Team members examine 
data together. The structure and content of this meeting will be used for 
each “Study” phase of the Plan-Do-Study-Act (PDSA) cycle. This meeting 
is titled “Meetings 5-8” because it is assumed that the Improvement 
Team will engage in three to four PDSA cycles; however, this meeting 
should be repeated for as many PDSA cycles as the Improvement Team 
launches, before moving to Meeting 9. In each of these data discussion 
meetings, team members will review data and determine whether to 
adopt, adapt, or abandon the change practice as well as decide on data 
collection strategies for the next PDSA cycle. 


Preparations 
Cl leader 


e Reach out to Improvement Team members partway through the 
“Do” phase to check in about data challenges and strategies, using 
the questions in Handout 20—Discussion protocol for data collection 
challenges and strategies (p. I-71). 


e Work with the data manager to ensure that PDSA cycle data are col- 
lected, summarized, and presented in an accessible way for discus- 


sion. This includes any use of graphs or charts. 


e Decide and confirm who will facilitate the data discussion. Will this be 


Materials needed 


Completed run charts and other 
summarized data from between 
meeting activity 
Plan-Do-Study-Act (PDSA) planning 
tool for this PDSA cycle (completed 
during Meeting 4 or revised at the 
end of a previous PDSA cycle) 
Pre-meeting reading: Data discus- 
sions and decisions (p. I-60) 
Handout 18—Protocol for data 
dialogue (p. II-65) 

Handout 19—Adopt, adapt, or 
abandon flow chart (p. Il-69 and 
https://ies.ed.gov/ncee/edlabs/ 


regions/northeast/pdf/handout19_ 
Adopt-adapt-abandon.pdf) 
Handout 20—Discussion protocol 


for data collection challenges and 
strategies (p. Il-71) 

Data from previous PDSA cycles, if 
this is not the first PDSA cycle for 
this change practice 


the same person in every data-related meeting? Will facilitation rotate among members? 


e Decide how data will be presented. Will data be presented on paper copies or projected on a screen for 
Improvement Team members to view. Will data be distributed in advance? 


e Ask Improvement Team members to complete the pre-meeting reading for Meeting 5 (see next page) on exam- 
ining data and making decisions based on results. 


Improvement Team members 

e Implement change practice and collect data as assigned. 

e Follow prescribed procedures for submitting data. 

¢ Complete the pre-meeting reading (see next page) on examining data and making decisions based on results. 

Data manager 

¢ Collaborate with the Cl leader to reach out to Improvement Team members partway through the “Do” phase 
to check in about data challenges and strategies, using the questions in Handout 20—Discussion protocol for 


data collection challenges and strategies (p. |I-71). 
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e Ensure that all data are collected, summarized, and presented in an accessible way for discussion (refer to 
Handout 17—Run chart templates and examples on p. I-51 and _ https://ies.ed.gov/ncee/edlabs/regions/ 
northeast/xls/handout17_Run-chart.xlsx). 


Recorder 


e Send the notes from Meeting 4 in a timely manner to all Improvement Team members and save the notes in a 
shared location that is accessible to all team members. 


e Prepare to take notes for the meeting. 
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Pre-meeting reading: Data discussions and decisions 


This pre-meeting reading will help Improvement Team members orient themselves to the processes, tools, and 
resources that will be used to support data discussion, analysis, and decisions in Meeting 5 and subsequent data 
dialogue meetings. 


Data discussions 


Handout 18—Protocol for data dialogue (p. \|-65) guides the Improvement Team through the process of observ- 
ing and analyzing the data. The protocol outlines five steps for holding a team data dialogue: 


Step 1. Review the purpose of the data discussion to create a shared understanding of the goals and norms for the 
discussion and identify any significant data limitations. It is not necessary to uncover every limitation; however, 
if serious concerns arise about the utility of the data for assessing the change practice, the Improvement Team 
could consider postponing the data discussion until additional data are collected. 


Step 2. Observe the data. Begin by describing the data with simple factual statements. Start with the simplest 
descriptions, attempting to avoid any inferences. Statements such as “10 students used sentence starters on Day 
1, and 11 students used sentence starters on Day 2” are appropriate. Often, these simple statements are neces- 
sary to ensure that Improvement Team members are reading the data accurately. 


Step 3. Make inferences based on the data. Inferences go beyond simple description to observations of the 
trends, patterns, outliers, and differences across subgroups revealed in the data. They might include statements 
such as “The number of students who used sentence starters increased from the first day to the second day” or 
“Students used two of the sentence starters much more than the others.” 


Step 4. Interpret the data. Use the data to better understand the change practice. What evidence do the data 
provide related to the change practice? Determine what can realistically be learned about the change practice 
from the data and what cannot be learned. In other words what do the data not tell you? Statements at this 
step may include, “Students seem more comfortable using the ‘I notice’ sentence starter than the ‘I agree/dis- 
agree with...’ sentence starter. This may be because their listening skills are still limited and that sentence starter 
requires them to listen to their peers in class.” 


Step 5. Conduct a pulse check on the data conversation. Conclude the data discussion with group reflection to 
“take the pulse of the group” related to the data discussion to identify what could be improved for future data 
discussions. 


Common challenges with data discussions 


The Improvement Team needs to be mindful of the challenges it may face during the discussion of the data and 
the many hazards that can derail the important process of interpreting and making sense of the data. These chal- 
lenges include: 


e¢ Moving too quickly to interpretation before fully examining the data. |n data dialogues people often move too 
quickly and draw conclusions based on experience and gut instincts without objectively considering the data 
at hand. By following the practice of describing the data before drawing conclusions, participants can put pre- 
conceived notions about what should or would happen to the side. A focus on data helps keep biases in check. 
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e Being overwhelmed by data. Excessive amounts of data can hinder progress, creating confusion and fatigue. 
An important role of the data manager is to prioritize and summarize data based on the Improvement Team’s 
focus for a given Plan-Do-Study-Act (PDSA) cycle, as described in the PDSA tool (completed during Meeting 4). 
Capturing the data that relate to the specific drivers and change practices will ensure that the most important 
data are presented in ways that support the development of decisions and action. The team can also consider 
eliminating data collection activities that are not yielding useful and actionable information. If in doubt, the 
data manager should consult with the Cl leader or other team members for input and guidance. 


e Interpreting variation in data. Whatever the change practice is, data are collected by different individuals oper- 
ating in different contexts. For example, did classrooms have different populations, such as one classroom 
with a higher percentage of English learner students than the others? Were some implementers of the change 
practice administrators while others were teachers? Did the change practice take place in classrooms or after 
school programs, or both? Collecting data on where and how the change practice was implemented will help 
the Improvement Team better understand why discrepancies exist and where or how the change practice 
might be more successful. 


Making decisions based on the data 


After each change practice is tested, the Improvement Team reviews the data collected and determines whether 
to adopt, adapt, or abandon the change practice. During this step the team makes decisions about modifying or 
fine-tuning the tested change. This is also when the team decides whether and how to scale up the change prac- 
tice or try it in different contexts. 


Use the predictions made on the PDSA planning tool for this cycle (completed during Meeting 4) to guide deci- 
sions. Results that are consistent with predictions may indicate that the change practice is having the intended 
outcome and represents an improvement toward the aim, and the Improvement Team may want to continue 
testing to confirm the results or eventually implement the change practice on a wider scale. Results that are not 
consistent with predictions can be the result of implementation issues, if the change practice was not imple- 
mented as intended; the quality or sensitivity of the measures, if the measure is not picking up on changes that 
members of the Improvement Team believe is occurring; or a change practice that will not yield improvements 
related to the aim. Further testing may be necessary. 


Handout 19—Adopt, adapt, or abandon flow chart (p. \|-69 and https://ies.ed.gov/ncee/edlabs/regions/northeast/ 
pdf/handout19_Adopt-adapt-abandon.pdf) will guide the Improvement Team at this stage. The flow chart will 
help the team reflect on how well it is implementing the change practice (through process measures), discern 
whether the change practice is resulting in improvement (through outcome measures), and assess whether the 
costs of the change practice are balanced against the benefits (through balancing measures). 


The Improvement Team will usually determine that it should move forward by adapting the change practice or 
how it is implemented or measured. It is recommended that the team refrain from adopting or abandoning a 
change practice until after multiple PDSA cycles. This is because the team will usually need multiple attempts 
to determine that the change practice has been consistently implemented with high fidelity, to consistently see 
good results, and to consistently assess that the costs are reasonable. The team will also need multiple cycles to 
assess whether the change practice is achieving the desired results and to determine whether a larger change is 
needed. 
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Critical questions before moving forward include: 


e Adopt: Should the change practice be tested on a larger scale? |f the Improvement Team saw actual improve- 
ment and positive movement toward the outcome, it may be time to implement the change practice ona larger 
scale, in more classrooms or schools. Several PDSA cycles are recommended before this option is selected. 


e Adapt: Does the change practice need to be adjusted? |\f adjustments are needed to the change practice, to 
a process related to implementing the change practice, or to data collection, the Improvement Team will 
make the changes and go through another PDSA cycle. This is the most common scenario. Repeating the cycle 
without making any changes to collect more data is also an option. 


e Abandon: Should the change practice be abandoned? \f the Improvement Team concludes that the change 
practice that was tested was implemented well but did not achieve the desired outcomes or that process or 
balance measures indicate that the approach is not likely to achieve the desired outcomes, it may be appro- 
priate to abandon a change practice. In that case, return to the driver diagram and consider whether a new 
change practice should be introduced and tested. 
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Meetings 5-8 
Goals 


e Review tools and processes for examining data (Meeting 5 only). 


Examine data (“Study”) collected during the “Do” phase of the PDSA cycle. 


Decide how to proceed in light of the data (“Act”). 
e Determine steps for the next PDSA cycle, if applicable (“Plan”). 


Agenda (90 minutes) 


Topic Materials Time 
Welcome, review of agenda and goals, and __| Toolkit, including a customized agenda; pre-meeting 15 minutes 
discussion of pre-meeting reading reading 

Watch and discuss Video 5 on data in Video 5 (https://youtu.be/jb7PkjMsxdw) 10 minutes 
continuous improvement (Meeting 5 only) 

Activity: Data dialogue Toolkit, Handout 18—Protocol! for data dialogue (p. ||-65) | 30 minutes 
Activity: Discuss the adopt, adapt, or Toolkit, Handout 19—Adopt, adapt, or abandon flow 15 minutes 
abandon flow chart chart (p. II-69 and https://ies.ed.gov/ncee/edlabs/ 


regions/northeast/pdf/handout19_Adopt-adapt- 
abandon.pdf) 


Activity: Discussion of data collection Handout 20—Discussion protocol for data collection 10 minutes 
challenges and strategies challenges and strategies (p. ||-71) 
Wrap up and next steps Toolkit, PDSA planning tool for this PDSA cycle 10 minutes 
(completed during Meeting 4 or revised at the end of a 
previous PDSA cycle) 


Welcome, review of agenda and goals, and discussion of pre-meeting reading (15 minutes) 


NKolKom Ko aal=\eiulatemi-lellin-laels 


Determine in advance who will facilitate the data discussion. This can be the meeting 
facilitator or someone else from the Improvement Team. The role of the data 
dialogue facilitator is to use Handout 18—Protocol for data dialogue to move the team 
through the steps of a productive data discussion. Before starting the discussion, the 
meeting facilitator or data dialogue facilitator should remind the team of the steps 

of the discussion, emphasizing the importance of observing the data before moving 
to interpretation and basing interpretations on the data. The data dialogue facilitator 
should also monitor participation and help engage everyone in the discussion. 


Begin the meeting with a welcome, a review of the agenda and goals, and a short discussion of the pre-meeting 
reading. Review: 


e The steps in the data dialogue. 
e Common data dialogue challenges. 


e Considerations for adopting, adapting, or abandoning a change practice. 
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Watch and discuss Video 5 on data in continuous improvement (Meeting 5 only) (10 minutes) 

Video 5 (available at https://youtu.be/jb7PkjMsxdw) shows the teachers and principal from Grove Elementary 
School reflecting on the data collection and data study aspects of their continuous improvement effort. After 
viewing the video, discuss: 

e What challenges the teachers described in collecting data. 

e What benefits the teachers saw in systematically collecting and reviewing data. 

Activity: Data dialogue (30 minutes) 

Present run charts and other summarized data so that all Improvement Team members can see them. Use the 
series of steps in Handout 18—Protocol for data dialogue (see next page) to examine and discuss the data collect- 


ed during the PDSA cycle. Summarize findings in the “Results” column of the PDSA planning tool for this PDSA 
cycle. 


I-64 


Handout 18. Protocol for data dialogue 


Step 1. Review the purpose of the discussion. Create a shared understanding of the goals and norms for the 
discussion and identify any significant data limitations. It is not necessary to uncover every limitation; however, 
if serious concerns arise about the usefulness of the data for assessing the change practice, consider postponing 
the data discussion until additional data are collected. 


Discussion prompts: 


e The purpose of this Plan-Do-Study-Act (PDSA) cycle was to test the following change practice: 


e Today we are discussing data from the following sources: 
e Were there significant problems that may have influenced the results? These may include problems with imple- 
menting the change practice or collecting data. Should we proceed with the discussion or postpone until these 


problems are resolved? 


Step 2. Observe the data. Describe the data with simple factual statements. Start with the simplest descriptions, 
attempting to avoid any inferences. 


Discussion prompts: 

e Describe the data. For example, “Teachers used sentence starters in 7 class periods during the 4-week cycle” 
or “Six students used sentence starters the first time we collected data and 14 students used sentence starters 
the final time we collected data in this cycle.” 


e Are you reading the data accurately? Is anything about the data unclear? 


Step 3. Make inferences. Observe trends, patterns, outliers, and differences across subgroups revealed in the 
data. 


Discussion prompts: 


e What patterns or lack of patterns do you see in the data? For example, “Students increased their use of sen- 
tence starters over the course of the cycle” or “Boys used the sentence starters more than girls.” 


Step 4. Interpret the data. Discuss how the data help you understand the change practice. What evidence do the 
data provide related to the change practice? Determine what can realistically be learned about the change prac- 
tice from the data and what cannot be learned. In other words, what do the data not tell you? 

Discussion prompts: 

¢ To what extent do the data provide evidence that the change practice is leading to the targeted outcomes? 

e How might variation in contexts explain any variation in results? 


e What hypotheses and assumptions do the data confirm or not confirm? 
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e What conclusions can we make, based on the data? 


e What are the limitations of the data and any conclusions we are making? Are there any contextual, implemen- 
tation, or data quality factors we should consider in making conclusions? 


e Do these data provide the evidence we need or do we need to collect additional/different data? What addi- 
tional data might we need? 


At the conclusion of this step, stop and record the results in the PDSA planning tool for this PDSA cycle. 

Step 5. Conduct a pulse check on the data conversation. Conclude the discussion with group reflection to “take 
the pulse of the group” related to the data discussion. This will help identify what could be improved for future 
data discussions. 

Discussion prompts: 


e Did the presentation of data make it easy to understand and interpret the data? 


e What might be helpful to change for next time, before, during, or after you look at the data, to help you decide 
to adopt, adapt, or abandon? 
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Activity: Discuss the adopt, adapt, or abandon flow chart (15 minutes) 


Use Handout 19—Adopt, adapt, or abandon flow chart (p. \l-69 and https://ies.ed.gov/ncee/edlabs/regions/ 
northeast/pdf/handout19_Adopt-adapt-abandon.pdf) to help decide how to proceed with the change practice. 
Follow the flow chart to answer questions about fidelity of implementation of the change practice and about 
evidence that the change practice is an improvement. After a single cycle the most likely decision will be to repeat 
the change practice, perhaps with adaptations. Employing an adopt, adapt, or abandon framework is a common 
approach to making decisions about what to do after one or more cycles of testing a change practice (Ortenblad, 
L6fstrom, & Sheaff, 2015). The flow chart is a guide to help the Improvement Team make decisions. But the team 
is not required to follow the direction of the flow chart. There may be circumstances in which a path different 
from what the flow chart outlines makes sense to pursue. 


Consider the following questions in assessing whether to adopt, adapt, or abandon the change practice: 


e Was the change practice implemented with fidelity? Reflect on whether the change practice was enacted as 
expected. High fidelity of implementation is required for confidence that any improvement is likely to be the 
result of the change practice. To assess fidelity of implementation, review the data collected through process 
measures from the PDSA planning tool for this PDSA cycle. If the process measures do not provide enough 
evidence to determine how well the change practice has been implemented, the Improvement Team automat- 
ically goes into the middle “Somewhat” path of the flow chart. 


e Is there evidence that the change practice is associated with improvement? lf the Improvement Team is moving 
forward with high fidelity, turn to whether the change practice is resulting in improvement and whether the 
improvement is balanced against its costs (the green boxes on the left). Assess these questions using the 
outcome measures and balancing measures from the PDSA planning tool for this PDSA cycle. If the evidence 
from these measures is positive—meaning the change practice is credibly associated with improvement and 
the costs are acceptable—and if the evidence is positive across multiple PDSA cycles, the team is ready to 
adopt this change practice. Move forward to Meeting 9 and decide whether to begin testing a new change 
practice or to work on scaling the change practice to a larger audience. 


e /Is there interest in trying to improve implementation? \f the Improvement Team is implementing the change 
practice with low fidelity, inconsistent fidelity, or no fidelity, discuss whether the team has the interest and 
capacity to engage in this change practice. The teams may feel pressured to pick change practices that people 
in positions of leadership think they “should be doing” or the team may be influenced by the sentiment “every- 
one is doing this, so we should too.” Selecting change practices without having an earnest desire to test them 
can result in poor implementation and a lack of progress. If the team is interested in testing the change prac- 
tice, keep moving forward with it. But if the team is not interested or capable, the best thing to do is to explic- 
itly abandon the idea, regroup, and select a new change practice backed by active interest. 


It is likely that after a single or even a few cycles the Improvement Team will follow Handout 19—Adopt, adapt, 
or abandon flow chart (p. \I-69 and https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/handout19_Adopt- 
adapt-abandon.pdf) and arrive at the recommendation to adapt the change practice and conduct another PDSA 
cycle. There are many different ways to adapt a change practice, and it is important to connect the correct type 
of adaptation to the pathway the team took through the flow chart: 


¢ Change measurement. If the Improvement Team reached the adapt recommendation because of any “not sure” 
pathways, consider adapting the work by changing the process, outcome, or balancing measures, as appropri- 
ate. Continuous improvement work is grounded in the collection, analysis, and review of evidence. The team 
cannot move forward in its work without sufficient evidence. 
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° Change procedures to improve implementation. If the Improvement Team reached the adapt recommendation 
because the change practice was too costly, consider streamlining it. Reducing the burden of implementation 
will improve both the fidelity of implementation and the balance of the improvement against the costs. 


e Redesign the change practice. If the Improvement Team is not seeing improvement from the change prac- 
tice, but implementation is occurring with fidelity, redesign the change practice. If evidence suggests that the 
change practice as it is currently designed will not result in the progress needed, make adjustments or modifi- 
cations to the practice. 


e Perform another iteration without adaptations. Doing another cycle as-is can be an appropriate adaptation. 
In fact, often several cycles of testing a change practice prior to adopting the practice is best, to confirm the 
consistency of positive results. The Improvement Team will not want to tinker with a change practice that is 
working well. It could also be the case that the team is getting used to the change practice and would like to 
improve implementation with another PDSA cycle. 


I-68 


Handout 19. Adopt, adapt, or abandon flow chart 


SREL 


shrine Same aCe (e) 8 mmr-Lel-] 0) er] ey-lale(ola miei mear-lat 


Regional Educational Laboratory 
AVEDC 


Was the change practice implemented with 
fidelity? 


Is there evidence that the : : : 
change practice is resulting in Is there interest in trying to 
improvement? improve implementation? 


(no) Have you implemented 
fewer than 3 PDSA cycles for (ves) 
Ges) this change practice with e— (no) 


similar results? 
Is change balanced (no) 
(costs acceptable)? 


Do you have evidence that 
further iteration will result in 
Fe5 improvement? 


Have you implemented two or 
more PDSA cycles for this 
change practice with similar 


results? 
ADAPT ABANDON 
e Perform another iteration without e Identify and begin the testing 
ADOPT adaptations process for anew change practice 


connected to a different driver 


e Identify and begin the testing 
process for anew change practice 


e Redesign change practice 

Revisit the driver diagram and 

e Change procedures to improve consider revising its design and/or 
implementation and/or balance brainstorming new change 

practices from the team’s PDSA 

e¢ Change measurement cycles 


e Begin scaling the change practice 
outside of the continuous 
improvement team 


This document is a part of the Continuous Improvement in Education Toolkit, produced by the Regional Educational Laboratory Northeast & Islands. 
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Activity: Discussion of data collection challenges and strategies (10 minutes) 


Handout 20—Discussion protocol for data collection challenges and strategies (see next page) guides discussion 
of what is working and what is most challenging in collecting data. Asking the questions before embarking on the 
next PDSA cycle will help the Improvement Team address any problems encountered in collecting and analyzing 
data in this cycle. It may also be valuable to check in with team members before the end of the cycle, through an 
email or short meeting, to discuss these questions and surface any issues. 
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Handout 20. Discussion protocol for data collection 
challenges and strategies 


e What data collection activities worked well in terms of time spent, data quality, and usefulness of results? 
e What data collection activities did not work well? 


¢ Do we need to address the feasibility of data collection for the Improvement Team or other practitioners? If 
so, how can we improve the feasibility of data collection, keeping in mind the goal of practical data collection? 


¢ Do we need to improve data quality? How can we do so? 


e What data collection strategies from this cycle should we use again next cycle? What new data collection strat- 
egies and activities should we use? 
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Wrap up and next steps (10 minutes) 

Plan for the next PDSA cycle or move to reflecting on multiple PDSA cycles (Meeting 9). To plan for the next PDSA 
cycle, refer to Handout 15—Template for the Plan-Do-Study-Act planning tool (p. \|-44 and https://ies.ed.gov/ 
ncee/edlabs/regions/northeast/pdf/handout15_PDSA-template.pdf) to: 


e Review the change practice, including any adaptations that will be made, and discuss what high-quality imple- 
mentation of the change practice entails to ensure a common understanding. 


e Record who is going to be testing the change practice and how often the change practice is going to be tested. 

e Review the data collection plan and assign responsibility for adjusting any of the measurement tools. 

e Set the next PDSA meeting date and time. 

Cl leader 

¢ Follow up with the Improvement Team members responsible for preparing for the next PDSA cycle, if applica- 
ble. This includes those who will prepare documentation about any adjustments to the change practice or to 
data collection tools or templates. 

e Disseminate the materials needed to implement the change practice, if applicable. 

¢ Communicate with implementers about the timing of change practice implementation and data collection. 
Check in regularly with implementers and data collectors to ensure that the change practice and data collec- 
tion are occurring as planned and to determine whether additional resources and supports are needed. 


¢ Work with the data manager to monitor data quality. 


e If the Improvement Team has conducted several PDSA cycles and is moving on to Meeting 9, compile data and 
documents from this series of PDSA cycles, organized chronologically. 


Improvement Team members 

e Implement the change practice. 

¢ Communicate with the meeting facilitator or the data manager about progress, challenges, and support needs. 

e Collect and submit data. 

e If the Improvement Team has conducted several PDSA cycles and is moving on to Meeting 9, review material 
provided and complete the pre-meeting reading on reflecting on multiple PDSA cycles. Come to the meeting 
prepared to discuss learning and next steps. 

Data manager 

e Make any necessary adjustments to tools or processes for collecting, compiling, or managing data. 


e Follow any procedures the Improvement Team has identified for monitoring and improving data quality. 
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e Collect, compile, and prepare data for the next Improvement Team discussion. 

e Ifthe Improvement Team has conducted several PDSA cycles and is moving on to Meeting 9, combine data and 
create run charts spanning all the PDSA cycles. If this is not feasible given changes in the measures, ensure that 
the individual cycle run charts are available and organized chronologically to allow team members to visualize 
a flow of time. 


Recorder 


¢ Document decisions made during the PDSA cycles and keep track of the completed PDSA planning tools from 
each cycle. 


e Send the notes from Meeting 5 in a timely manner to all Improvement Team members and save the notes ina 
shared location that is accessible to all team members. 


e Prepare to take notes for Meeting 9. 
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Meeting 9. Reflecting across multiple Plan-Do-Study-Act cycles 


The purpose of this meeting is to reflect across several Plan-Do-Study- 
Act (PDSA) cycles and synthesize the data that the Improvement Team 
has collected and the lessons learned across several tests of one or 
more change practices. The team will also decide how to proceed with 
the continuous improvement effort. The correct timing for this meeting 
depends on the team’s schedule and progress. For example, at one 
school the team may run short two-week PDSA cycles. In that case, after 
three months the team will have conducted six PDSA cycles, probably 
implementing more than one change practice. Another team may run 
month-long PDSA cycles and, over the course of three months, have 
conducted three PDSA cycles, all focused on adaptations of a single 
change practice. At a minimum the team should conduct at least three 
PDSA cycles before this meeting since the purpose of this meeting is to 
reflect on the overall value of the change practice and determine the 
team’s next steps. It is important to have several cycles completed to 
make any clear observations about the change practice and the status of 
the team’s work. 


Preparations 


Cl leader 


Materials needed 


Completed Plan-Do-Study-Act 
(PDSA) planning tools from each 
PDSA cycle 

Run charts and other data collect- 
ed during the PDSA cycles 
Documentation of decisions made 
during PDSA cycles (from adopt, 
adapt, or abandon flow chart and 
meeting notes) 

Completed driver diagram (from 
previous meetings) 

Pre-meeting reading: Reflecting on 


multiple Plan-Do-Study-Act cycles 


Handout 21—Discussion questions 
for reflecting on multiple Plan-Do- 
Study-Act cycles (p. II-78) 


e Work with the data manager to compile data and documents from this series of PDSA cycles, organized chrono- 
logically. This includes the completed PDSA planning tools for each PDSA cycle, additional meeting notes, and 
run charts for the cycles. 


e Ask Improvement Team members to complete the pre-meeting reading (see next page) on reflecting on multi- 
ple PDSA cycles. 


Improvement Team members 


e Review material provided and complete the pre-meeting reading (see next page) on reflecting on multiple 
PDSA cycles. 


¢ Come to the meeting prepared to discuss your learning and next steps. 

Data manager 

¢ Combine data and create run charts spanning all the PDSA cycles. If this is not feasible given changes in the 
measures, ensure that the individual cycle run charts are available and organized chronologically to allow 
Improvement Team members to visualize a flow of time. 

Recorder 


e Share documentation of decisions made during the PDSA cycles with the meeting facilitator and Cl leader. 


e Prepare to take notes for the meeting. 
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Pre-meeting reading: Reflecting on multiple Plan-Do-Study- 
Act cycles 

Review work to date 

In preparation for the cross-cycle reflection meeting, review these materials: 

¢ Completed Plan-Do-Study-Act (PDSA) planning tool(s) reflecting the work for each cycle. 

¢ Completed run charts. 

e Additional meeting notes or other decision tracking documentation. 


Synthesize learning 


The goal in reviewing all these materials is to take a broad view of the information contained in the documents 
and to find connections, comparisons, or contrasts between the PDSA cycles. The Improvement Team will identify 
the lessons learned that go beyond what any individual cycle would teach on its own. The three guiding questions 


from the Model for Improvement should guide the reflection and conversation: 


What problem are we trying to solve? The Improvement Team developed the driver diagram, with an overar- 
ching aim statement, early in the continuous improvement effort. The aim statement focused work and built a 
shared understanding of the team’s ultimate goal. The aim is purposefully documented at the top of the PDSA 
planning tool for each cycle to continually orient the team toward this goal. Depending on the nature of the 
aim statement, the team may or may not have collected data directly related to the aim statement that it can 
refer to at this point. For example, an aim statement related to increasing classroom participation may have 
data associated with it and therefore the team can evaluate progress toward the aim during each PDSA cycle, 
whereas an aim statement related to annual state tests would not have related data collected as part of the 
PDSA cycles. Rather, the team would focus on outcome data related to the drivers, which the team has hypoth- 
esized are related to the overall aim of improving student achievement. 


What change might we introduce and why? This is one of the core questions that the Improvement Team 
explored in initial meetings as it mapped out the fishbone and the driver diagrams. The team selected change 
practices that members expected would contribute to improvements. Then the team made decisions in each 
cycle about whether to adopt, adapt, or abandon the change practice. 


How will we know that a change is an improvement? This is the core question of Meetings 5-8 and guides the 
“Study” phase of each PDSA cycle. The Improvement Team’s task with each cycle is to review and reflect on the 
data collected to evaluate this question for each test of change practice, whether it is a new change practice or 
an adaptation of a prior change practice. During this reflection meeting the team’s goal is to consider whether 
the change practice has resulted in improvement toward the aim. To do so, it must consider whether data from 
several PDSA cycles indicate that improvement is occurring. 


Deciding whether to implement a change practice 


One goal for this reflection meeting is to decide whether to conduct additional PDSA cycles or to transition efforts 
toward implementing the tested change practices as usual practice in the school or district. The team should 
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decide together whether there is enough evidence of improvement to transition from the testing phase to the 
implementation phase of the continuous improvement work. 


There are several considerations to think through in making this decision: 


e Progress toward the aim statement. Sometimes a single well implemented change practice will result in sig- 
nificant improvement on its own. In that case implementing a single change practice could be a worthwhile 
endeavor. However, the Improvement Team often determines that a single helpful change practice is insuf- 
ficient to make the needed progress toward the aim. In that case the team may find it necessary to further 
develop a “change packet” of multiple change practices before working on transitioning to implementing the 
change practice. 


e Understanding of the change practice or change packet. Implementing a change practice or change packet 
requires the Improvement Team to understand what the change practice or change packet is, when to use it, 
how to maximize its value, and why it is resulting in improvement. Even with evidence that a change practice 
or change packet is associated with significant improvement, it is important to assess whether the team is 
prepared to train and support greater implementation. If not, another cycle focused on developing this level of 
understanding may be needed before implementing the change practices. 


e Schedule. How does the work the Improvement Team has done to identify, implement, and study change prac- 
tices to date fit into the larger context? Most school-based teams align their PDSA cycles with the academic 
year, academic semester, or some other significant scheduling consideration. Given the current date, assess 
whether the team has time to conduct additional tests of change practice. If not, it may be best to redirect 
efforts to documenting the team’s existing knowledge for use in the next semester or next year, even if the 
team has not yet fully realized the aim statement. 
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Meeting 9 

Goals 

e Review the findings from the multiple PDSA cycles. 

e Synthesize data and decisions. 

e Decide how to proceed with the continuous improvement effort. 


Agenda (90 minutes) 


Topic Materials Time 
Welcome and review of agenda and goals 10 minutes 


Activity: Review work to date and reflect | Completed PDSA tools, run charts, decision documentation, | 45 minutes 
on multiple PDSA cycles and driver diagram from each PDSA cycle; Handout 21— 
Discussion questions for reflecting on multiple Plan-Do- 
Study-Act cycles (p. \|-78) 


Activity: Decide whether to implement a 25 minutes 
change practice 


Wrap up and next steps 10 minutes 


Welcome and review of agenda and goals (10 minutes) 


Begin the meeting with a welcome and a review of the agenda and goals. Ensure that everyone has the appropri- 
ate materials for the meeting. Give Improvement Team members a few minutes to review the materials before 
launching into the first activity. 


Activity: Review work to date and reflect on multiple PDSA cycles (45 minutes) 


NKolKom Kol aale\eiulatcai-lelliv-leels 


Support the Improvement Team in keeping the discussion focused on evidence and 
the documents compiled for this meeting. If participants are focused on sharing 
opinions or inferences without explicit connections to the team’s work, use some of 
the following questions to refocus the conversation: 

e Where are you seeing that in our data? 

e Can you show me what is leading you to that conclusion? 

e Are you sure? How could we check that? 


In preparation for this meeting, Improvement Team members should have reviewed the following materials: 
¢ Completed PDSA planning tools for each cycle. 

¢ Completed run charts. 

e Additional meeting notes or other decision tracking documentation. 


Step 1. Spend a few minutes independently reviewing the collected work to date, focusing on what the change 
practice was, what evidence the Improvement Team collected, and what decisions the team made. 


Step 2. As a team, work through Handout 21—Discussion questions for reflecting on multiple Plan-Do-Study-Act 
cycles (see next page). 
11-77 


Handout 21. Discussion questions for reflecting on multiple 
Plan-Do-Study-Act cycles 


Discuss the following questions in whatever order makes the most sense to the Improvement Team. The team 
may bounce back and forth among several questions. Some of the questions may be more relevant than others, 
given the results of the Plan-Do-Study-Act (PDSA) cycles. 


e Is the Improvement Team making progress toward its aim statement based on the evidence collected thus far? 
Consider whether the outcome measures for each PDSA cycle are providing meaningful evidence that might 
suggest progress toward the overall aim. For example, if the team is collecting measures of student engage- 
ment in math class and sees improvement in student engagement, does the team believe this progress is the 
result of the change practice? 


e Does the Improvement Team see improvement across cycles that it didn’t see when looking at a single cycle? 
Consider a team that tested and adapted a change practice across four PDSA cycles. Each individual cycle may 
not have resulted in much change at all, perhaps improving student participation from 50 percent to 56 percent 
in the first PDSA cycle, from 56 percent to 62 percent in the second PDSA cycle, and so forth. However, across 
four cycles the improvement appears to jump from 50 percent to 74 percent. This tells a story about improve- 
ment that is very different from that told by any one PDSA cycle on its own. 


e Does the driver diagram contain the best change practices to make improvements toward the aim? Now that 
the Improvement Team has been through several PDSA cycles, it may have new ideas about the change prac- 
tice inspired by what it learned in implementing the change practice thus far. The team may also want to con- 
sider conducting additional searches for effective evidence-based practices. 


e Is the Improvement Team working on the right drivers? |f the team is not seeing significant progress toward the 
aim, it could reflect on which drivers have been the focus of the effort and consider whether another driver 
might lead to a more effective approach. 


e Does the driver diagram need to be revised? |f the Improvement Team has evidence from the prior cycles sug- 
gesting that some of the primary or secondary drivers are being realized but are not creating progress toward 
the aim, the team may want to revisit and revise the driver diagram to remove some of those drivers and 
include new drivers that it believes are more directly related to the aim. 


e Based on the evidence, what has the Improvement Team learned that might be relevant to other change prac- 
tices or adaptations? What has the team learned through the PDSA cycles that will inform choices about what 
to test next, how to effectively implement a change practice, or what data are most feasible and relevant to 
collect? 


e Does the Improvement Team need to change the outcome, process, or balancing measures? If the team is not 


seeing meaningful or actionable data or if the team has difficulty synthesizing knowledge across multiple PDSA 
cycles, changes to the measures may be warranted. 
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Activity: Decide whether to implement a change practice (25 minutes) 


Step 1. As a team, discuss the following questions: 


If other people received similar results to what we are seeing, would we consider that “good enough” progress? 


How much more do we think we can learn with another series of PDSA cycles? 


Do we understand what we’re doing well enough to train other people to do it? 


What external factors, such as scheduling, might influence our ability to continue testing or to implement, or 
both? 


Step 2. As a team, decide whether to conduct another series of PDSA cycles, to revisit the driver diagram to revise 
drivers or change practices, or to transition to implementing the change practice. 


At this point, the Improvement Team will confirm the next step for the improvement effort. 


e If the team decides to revisit the driver diagram, go back to Meeting 3 to prepare for revisions or develop a 
new driver diagram. 


e If the team decides to engage in another series of PDSA cycles, go back to Meeting 4 to prepare for the next 
test of change practice. 


e If the team decides to transition to implementing the change practice or a set of change practices, go to 
Meeting 10. 


Regardless of which path the team takes, be sure to celebrate your progress before concluding Meeting 9. 

Wrap up and next steps (10 minutes) 

Celebrate progress 

It is important in this reflective meeting to allocate time for acknowledging and validating the progress that the 
Improvement Team has made. Even if the team has hit some stumbling blocks, there is always something that can 
be gleaned from the work. 

Give all participants a chance to contribute to the discussion. First allow a few minutes for quiet reflection; then 
invite Improvement Team members to share their thoughts. Consider using the following sentence starters (or 
use your own): 

e |used to think... and now | think... 

e |am most proud of... 

e |/We have been challenged by... and are overcoming it by... 


Adding “because of...” to any of the above pushes the reflection deeper. 


Adding “as evidenced by...” to any of the above connects your larger experience to your improvement work. 
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Next steps 

The next steps will depend on the path the Improvement Team will make. If the team decides to revisit the driver 

diagram, go back to Meeting 3 for pre-meeting activities and readings. If the team decides to implement another 

series of PDSA cycles, go back to Meeting 4 to review the pre-meeting activities and reading. If the team decides 

to transition to implementing the change practice or a set of change practices, the next steps are provided below. 

Cl leader 

e Seek out information and input from the appropriate stakeholders to understand when, by whom, and where 
the change practice or change packet will be implemented. Consider how much time will be required to docu- 


ment, train, and prepare for the implementation of the change practice. 


e Ask Improvement Team members to complete the pre-meeting reading for Meeting 10 on preparing for imple- 
mentation of a change practice. 


Improvement Team members 

¢ Complete the pre-meeting reading for Meeting 10 on implementation of a change practice. 

e Reflect on and be prepared to discuss details of the change practice that will need to be communicated to new 
implementers. Also reflect on the continuous improvement effort more generally and be prepared to discuss 
what has worked well and what could be improved. 


Data manager 


e While the data manager will not have any designated tasks before Meeting 10, it is wise to be sure all data are 
in order and all data collection and storing processes have been documented. 


Recorder 


e Send the notes from Meeting 9 in a timely manner to all Improvement Team members and save the notes ina 
shared location that is accessible to all team members. 


e Prepare to take notes for Meeting 10. 
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Meeting 10. Preparing to implement a change practice more 


broadly 


The purpose of this meeting is to begin planning for implementation of 
the change practice so that it becomes business as usual in the intended 
setting.* Although the Improvement Team may not be responsible for 
supporting full implementation of the change practice, the team does 
have a responsibility to take what has been learned about the change 
practice through the Plan-Do-Study-Act (PDSA) cycles and help the 
school or district prepare for further or full implementation. 


Preparations 
Cl leader 


e Bring information collected from stakeholders about when, by whom, 
and where the change practice will be implemented. 


e Ask Improvement Team members to complete the pre-meeting 
reading (see next page) on preparing for implementation of a change 


practice. 


Improvement Team members 


Materials needed 


Existing documentation of the 
change practice, such as descrip- 
tion of practices or procedures 
Pre-meeting reading: Planning for 
scale up (p. II-82) 

Handout 22—Template for outlin- 
ing a change practice (p. II-85) 
Handout 23—Template for commu- 
nicating about a change practice 
(p. II-87) 

Handout 24—Completed template 


for communicating about a change 


practice for the Grove Elementary 
School example case (p. I|-88) 
Handout 25—Template for imple- 
mentation planning (p. II-91) 
Handout 26—Protocol for reflecting 
on the continuous improvement 
effort (p. II-92) 


¢ Complete the pre-meeting reading (see next page) on preparing for implementation of a change practice. 


e Reflect on and be prepared to discuss details of the change practice that will need to be communicated to new 
implementers. Also reflect on the continuous improvement effort more generally and be prepared to discuss 


what has worked well and what could be improved. 
Data manager 
e None. 


Recorder 


e Send the notes from Meeting 9 in a timely manner to all Improvement Team members and save the notes ina 


shared location that is accessible to all team members. 


e Prepare to take notes for the meeting. 


2. The documentation for this meeting refers to a single change practice, but it applies equally to a series of change practices (a “change 


packet”). 
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Pre-meeting reading: Planning for scale up 


The purpose of the Plan-Do-Study-Act (PDSA) cycles (Meetings 4—8) is to test change practices and determine 
whether they are resulting in improvement. Once the Improvement Team concludes that a change practice? is 
an improvement toward the overall aim and has decided to implement it as usual practice (Meeting 9), the focus 
shifts to planning for implementation. Successful implementation requires the team to formalize and document 
the change practice and develop or identify the supports and resources needed for initiating and sustaining 
implementation. 


Formalizing the change practice 


As Improvement Team members think about involving additional individuals and sites in implementing a change 
practice, they should use what they have learned from the PDSA cycles in answering: 


e What are the individual steps that make up the change practice, and how do they work? Practitioners will not 
be successful in implementing the change practice without a clear explanation of what the specific steps are. 
Written descriptions should include the essential components of the change practice with operational defini- 
tions (including acceptable and unacceptable variations of the practice), steps, and procedures. 


e Who will be involved in implementing the change practice? The Improvement Team should be specific about 
which people are taking which actions. This may be a simple question for the team if, for example, all math 
teachers are enacting the change practice in the same way. In other cases different groups of actors may need 
to take different actions, and the team should make this clear. 


e In what contexts and for what populations is the change practice intended? Users of the practice will need 
information about the contexts in which the practice should be implemented and the intended population. For 
example, it may be implemented during math instruction for all grade 5 students or for students who are not 
reaching specified benchmarks. 


The amount and type of documentation needed will depend on the complexity and nature of the change practice. 
Preparing for successful implementation of a change practice 

Changing adult behaviors in complex organizations such as schools and districts is not easy. Any change practice 
proposed by the Improvement Team, simple or complex, will require changes in the behaviors of individuals and 


possibly in how the school or district operates. The team can play a critical role in preparing for successful imple- 
mentation. The following strategies can help build a strong foundation for implementing the change: 


Gaining leadership support. Gain leadership support and commitment in order to achieve successful 
implementation. 


© Communicating about the change practice. Effectively communicate about the change practice to those who 
will be involved or affected. This is important for building buy-in. 


e Building capacity through training and materials. Plan for the types of training and materials that will support 
implementation. Even simple changes will require new knowledge and capacities. 


3. This discussion refers to a single change practice, but it applies equally to a series of change practices (a “change packet”). 
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e Identifying resources and infrastructure supports. Consider the materials and resources that new implementers 
will need, along with any implications for staffing, scheduling, and communication. 


e Monitoring progress to identify and address challenges. Set up a process to monitor implementation. This is 
essential if the goal is to achieve high-quality and sustained implementation. Implementation drift and aban- 
donment are normal occurrences in environments where many demands compete for staff attention. 


Reflecting on the continuous improvement effort 

Working together as an Improvement Team has almost certainly been a new experience for everyone. Contin- 
uous improvement is an approach to enacting change that is very different from the traditional ways in which 
schools operate. The team has probably encountered successes and challenges in working together that may 


inform future continuous improvement work. 


Each member of the Improvement Team should set aside time before Meeting 10 to reflect on the experience 
throughout the continuous improvement effort and identify what has worked well and what could be improved. 
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Meeting 10 


Goals 
e Formalize change ideas. 
¢ Begin planning for implementing the change. 


e Reflect on continuous improvement process. 


Agenda (90 minutes) 


Topic Materials Time 


Welcome, review of agenda and goals, 15 minutes 
and discussion of pre-meeting reading 


Activity: Formalizing the change practice | Handout 22—Template for outlining a change practice (p. \I-85) | 30 minutes 


Activity: Planning for implementation | Handout 23—Template for communicating about a change 30 minutes 
practice (p. II-87), Handout 24—Completed template for 
communicating about a change practice for the Grove 
Elementary School example case (p. |I-88), Handout 25— 
Template for implementation planning (p. \|-91) 


Wrap up and next steps Handout 26—Protocol for reflecting on the continuous 15 minutes 
improvement effort (p. ||-92) 


Welcome, review of agenda and goals, and discussion of pre-meeting reading (15 minutes) 


Begin the meeting with a welcome and a review the meeting goals and agenda. Then, briefly discuss: 
° Questions that need to be addressed to formalize the change practice. 


e Preparation needed to support the change practice. 
Activity: Formalizing the change practice (30 minutes) 


The Improvement Team’s first action in this final meeting is to begin formalizing the change practice. The goal 
is to be as detailed as possible, enabling people who have never seen the change practice before to understand 
what it is, why they should implement it, and what successful implementation of the change practice looks like. 


Use Handout 22—Template for outlining a change practice (see next page) to begin documenting the steps and 
practices for the change practice. If a series of change practices is being implemented, repeat the process for each 
change practice. The documentation is intended to provide all the information that practitioners need to imple- 
ment the change practice and for the school or district to be able to support implementation through training and 
supervision. The documentation may include information about what the change practice is, how and in what con- 
texts it should be implemented, and what resources and supports will be needed. Use information collected during 
the PDSA cycles to provide guidance on what works well and what to avoid in implementing the change practice. 


Review Handout 23—Template for communicating about a change practice (p. I|-87). If a series of change prac- 
tices is being implemented, complete a template for each change practice. The template can be used to introduce 
potential implementers and other stakeholders to the practice. See Handout 24—Completed template for com- 
municating about a change practice for the Grove Elementary School example case (p. ||-88) for guidance. 


The Improvement Team is unlikely to complete Handout 22—Template for outlining a change practice (p. \|-85) 
or Handout 23—Template for communicating about a change practice (p. \I-87) during Meeting 10. Rather, the 
team will likely need to identify next steps for their completion after the meeting. 
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Handout 22. Template for outlining a change practice 
Use the following outline to formalize the tested change practices. 
¢ Name of the change practice: 
e Purpose of the change practice: 
a. The Improvement Team was created to [overall focus]. 
b. Specifically, the goal was to improve [aim/outcome]. 


c. From testing, we know that [change practice] will improve 
[outcome] by [primary or secondary driver]. 


e Description of the change practice 


a. The major parts of the change practice are: [list the major components] 


b. The list of people who will be enacting the change practice are: 


a [person or group] will do [component] 
a [person or group] will do [component] 
a [person or group] will do [component] 


e Step-by-step details of the change practice 


a. First step of the change practice 


m The first specific step is: [small task]. 
m This looks like: [example]. 
m This could look like: [acceptable adaptation]. 
m This should not look like: [adaptation that failed in testing]. 


b. [Repeat for all steps of the change practice] 
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e Tips for implementing the change practice 
a. List of tests with positive results 


m When doing 
[positive recommendation] 


m [Repeat for more recommendations] 
b. List of tests with negative results 


m When doing 
[negative recommendation] 


m [Repeat for more recommendations] 
e Potential challenges and how to address them 


a. You might notice 
[solution]. 


b. [Repeat for more challenges] 
e Assessing implementation 
a. First measurement 
m When you start to implement 
measure 
measure because 


m You ideally want to see 


b. [Repeat for more measurements] 


[step #] you SHOULD 


[step #] you should NOT 


[challenge]. If so, you should try 


[change practice], you should 


[measurement] 


[measurement] is important to 
[rationale connected to effectiveness] 


[measurement result] because 
[rationale connected to target outcome] 


1-86 


Handout 23. Template for communicating about a change 
practice 


WHAT is the change practice? Briefly describe the practice being implemented. 


WHO will be using the change practice and WHEN will they start? WHO is going to support these people in using 
the change practice and HOW will they get support? 


WHY are we doing this? 


EXPLAIN the change practice. Provide examples. 


DO these things when implementing the change practice. 


AVOID these things when implementing the change practice. 
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Handout 24. Completed template for communicating about 
a change practice for the Grove Elementary School example 
case 


WHAT is the change practice? Briefly describe the practice being implemented. 


Teachers will use sentence starters at least twice a week in their math lessons to support more students 
talking and writing about math in the classroom. 


Teachers will use sentence starters to help students both talk and write about math. 


WHO will be using the change practice and WHEN will they start? WHO is going to support these people in using 
the change practice and HOW will they get support? 


The Grove Elementary School grade 4 and 5 teaching teams are going to start using the sentence starters 
in math lessons in the beginning of the coming school year. Once math sentence starters are being used 
routinely and effectively at these two grade levels, we will start using them in lower grade levels. The goal 
is for all teachers in Grove Elementary School to be using math sentence starters by the end of the first 
semester. 


The local experts on math sentence starters are Joanie Rivera (grade 4), Rosanna Barron (grade 5), and Toni 
Rhodes (math coach). Joanie and Rosanna are going to lead the grade-level Professional Learning Communi- 
ty meetings, along with Toni, to train teachers on how to create and use sentence starters in math lessons. 
They will provide some examples they have used themselves and will conduct a model lesson activity to 
demonstrate what has worked well for them in their classrooms. Toni will visit each teacher’s classroom at 
least once in September to observe and coach teachers on the use of sentence starters. 


WHY are we doing this? 


Our students at Grove Elementary School struggled with the new Common Core math curriculum that we 
adopted last year, and this was reflected in many students, particularly our low-income students, perform- 
ing below proficient on the state assessment. As we discussed at our whole-school teacher meeting at the 
beginning of this school year, our students are doing well at quantitative reasoning, but they have been 
struggling with talking and writing about their thinking. 


Over the last year a small team of teachers and staff worked together to address this problem: Trent Masters 
(grade 3), Joanie Rivera (grade 4), Rosanna Barron (grade 5), Toni Rhodes (math coach), and Rebecca Arm- 
strong (literacy coach). They started testing out different math sentence starters in November to see what 
would work best for our students. After spending the school year trying different approaches to math sen- 
tence starters, the Improvement Team feels ready to train others on best practices for how to incorporate 
these into math lessons on a regular basis. 
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EXPLAIN the change practice. Provide examples. 


Consider these questions about students’ skills: 


Are students unpacking their own thinking? 
Are students checking for understanding of what other students said? 
Are students actively comparing and contrasting their work with that of other students? 


Are students selecting appropriate math vocabulary? 


Notice that the four sentence starters below encompass a broad spectrum of student skills. Using different 
sentence starters will keep students more engaged than they might be if they always use the same one. 


Focused on strategy (Strategy) 

“The strategy | used was ; my strategy is like or different from because 
Focused on listening to others and repeating or restating (Repeat/Restate) 

“You’re saying that [repeat what said]; would someone like to add on?” 
Individual/initial thinking (Initial thinking) 

“\think ss because ; | believe this istrue because _———. Isolvedtheproblemby __—s«.” 
Student prompted to repeat prior student’s thinking and then provide individual thinking (Prompts/Vocab) 


” 


“The correct math vocabulary word is because it means 


DO these things when implementing the change practice. 


Introduce students to sentence starters in math orally at first. Students took a few days to warm up to 
talking about math and were not successful writing about their math thinking in the first few days after 
the sentence starters were introduced. 


Use sentence starters with partner activities to further support greater math discourse in your class- 
room. Partner activities force more students to spend more time talking. 


Circulate with purpose during partner activities. It is better to observe a few pairs of students than try 
to get to all pairs. In particular, check in on your English learner students to see whether they need addi- 
tional support or encouragement. 


Write sentence starters on the white board with large spaces where students are supposed to insert 
their own words or numbers. During whole-class discussion, write down what different students are 
saying using the sentence starters so that students can both see and hear others’ thinking. 
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AVOID these things when implementing the change practice. 


e Rushing. Don’t rush! You’ll be tempted to get students to use all the sentence starters right away, but stu- 
dents will take longer to get used to them than you do. Give yourself twice as much time as you think it will 
take at first until you get a better sense of how quickly your students are adapting to the sentence starters. 


¢ Calling only on students whose hands are up. The goal of this practice is to support more students in 
talking more often. You can print out a copy of your class roster and track how many times students have 
talked to help you better distribute whom you call on. Teachers who used participation sticks or other 
cold-calling techniques found that those techniques worked very well with math sentence starters. 


Activity: Planning for implementation (30 minutes) 


It is now time for the team to take what it has learned and begin planning for implementation. Implementation at 
this stage could be onboarding more teachers in a school to incorporate a new teaching practice or set of teach- 
ing practices, onboarding more schools in the district, or something else. 


Implementing the change practice is going to be a lengthy process that spans beyond the scope of this toolkit. 
The purpose of Meeting 10 is to begin planning the implementation process and transition the work to those 
responsible for full implementation, which may or may not involve the Improvement Team. 


To begin this transition, consider the following questions: 


e Who else needs to be included in the implementation planning process? If the work will branch out to other 
grade levels, departments, or schools, include teacher leaders or school leaders as appropriate for their buy-in 
and input. 


e When are suitable times to begin implementing a schoolwide or districtwide change practice or set of practic- 
es? Small teams of motivated people can often begin changes rather flexibly, whereas larger groups of people 
often require more structured, formal timing to begin something new. 


e What are the dependencies in the implementation planning? Map out which activities need to happen first, 
which activities need to happen second, and so forth. 


e Are there any anticipated barriers or resistance to implementation, and if so, what are they? The Improvement 
Team reflected on barriers to change they expected to encounter in beginning the continuous improvement 
work. As the team prepares for the communication and implementation phase of the work, it is important to 
work through this process again. 


Use Handout 25—Template for implementation planning (p. \|-91) to begin identifying the major activities that 
will need to be conducted to support further implementation. As with the other activities in this meeting, it is 
unlikely that the template will be completed in Meeting 10. Use this time to begin planning and identify next steps 
for the implementation process. 


Wrap up and next steps (15 minutes) 
Reflecting on the continuous improvement effort 
Use Handout 26—Protocol for reflecting on the continuous improvement effort (p. II-92) to reflect on the contin- 


uous improvement effort and identify improvements for future work. 
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Handout 25. Template for implementation planning 


Use the following planning tool to identify the supports and activities that will be needed to implement and 
sustain the change practice. 


Tan}e)(=vaatelale-incedam-(eun\ ing Specific tasks and timeline Who is responsible? 


Document (formalize) the practice. 
Include essential components, 
operational definitions, steps and 
procedures, and contexts in which the 
practice should be implemented. 


Leadership support. Work with 
administrators and other key leaders 
to ensure that they understand and are 
committed to the change. 


Communication. Develop a plan to 
communicate to practitioners, support 
staff, and other stakeholders (such as 
families or students) the purpose of the 
innovation, how it works, and where to 
obtain further information and support. 


Training. Develop a plan for materials 
(for example, presentation slides, 
handouts, or agenda), staffing (in house 
or external), timing, and how training 
will be sustained as staff turnover. 


Resources. Identify what will be needed 
for implementation, such as copies of 
written materials, instructional supplies 
or equipment, and release time for 
training attendance. 


Support. Indicate who will serve 

as the experts to be a resource to 
implementers. Develop a plan for how 
these individuals will communicate 
with and check in with implementers, 
especially at the beginning of 
implementation, to ensure that 
implementation is going smoothly. 


Monitoring. Develop a plan to 
determine whether the innovation is 
being implemented as intended and 
whether additional support is needed 
for successful implementation. 


Other supports. Determine whether 
anything further is needed to facilitate 
successful implementation. 
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Handout 26. Protocol for reflecting on the continuous 
improvement effort 


Reflect on the following elements of the continuous improvement work. 


What worked well? NAV/at-lexerol¥] (ol oX-m lanl elnec=r0 ra 


Tools and resources 
(Fishbone diagram, Driver 
diagram, Plan-Do-Study-Act 
(PDSA) planning tool, Protocol 
for data dialogue, others) 


Meetings 
(frequency, duration, structure) 


Identifying and implementing 
change ideas 


Collecting and analyzing data 


Data dialogue and making 
decisions based on data 


Other aspects of the continuous 
improvement effort 


e Overall, what have been the primary benefits of the continuous improvement effort to your school or district? 


e Overall, what have been the primary costs or negative aspects of the continuous improvement effort for your 
school or district? 


e What changes should be made to improve continuous improvement efforts at your school or district in the 
future? 
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Next steps 

Once the Improvement Team has decided to implement a change practice (Meeting 9) and begun preparing for 
implementation (Meeting 10), it is time to transition the work to those who will be responsible for full implemen- 
tation. Complete the steps below to conclude the Improvement Team’s work. 


Cl leader 


e Assemble the work conducted during the meeting on the change practice outline, the change practice commu- 
nication guide, and the implementation planning and identify next steps for their completion. 


¢ Contact the individual or individuals responsible for leading full implementation and share documents, such as 
those described above, that are needed for implementation. 


e Ensure that documentation from the Improvement Team’s work is in order and available for future continuous 
improvement work. 


Improvement Team members 


e Assist the Cl leader as necessary in identifying next steps and sharing information with those who will be 
responsible for full implementation of the change practice. 


¢ Be available to provide additional information and support those responsible for implementing the change. 
Data manager 

e Ensure that all data are in order and that data collection and storing processes have been documented. 
Recorder 


e Send the notes from Meeting 10 in a timely manner to all Improvement Team members and save the notes ina 
shared location that is accessible to all team members. 
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